Sep 15 21:15:51.670747 ip-10-0-129-100 systemd[1]: Starting Kubernetes Kubelet... Sep 15 21:15:52.081811 ip-10-0-129-100 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:52.081811 ip-10-0-129-100 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 21:15:52.081811 ip-10-0-129-100 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:52.081811 ip-10-0-129-100 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 21:15:52.081811 ip-10-0-129-100 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 21:15:52.084255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.084171 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 21:15:52.088281 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088265 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.088281 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088281 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088284 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088287 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088291 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088294 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088298 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088301 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088305 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088308 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088311 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088313 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088316 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088319 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088321 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088324 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088327 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088329 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088332 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088334 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.088342 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088337 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088340 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088342 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088346 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088349 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088352 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088355 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088358 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088360 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088363 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088366 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088368 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088371 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088373 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088376 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088378 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088381 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088383 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088387 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088389 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.088806 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088394 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088397 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088400 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088402 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088405 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088409 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088413 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088416 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088419 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088422 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088424 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088427 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088431 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088433 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088437 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088440 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088443 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088446 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088449 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.089283 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088462 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088466 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088468 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088472 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088474 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088477 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088480 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088482 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088485 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088488 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088490 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088493 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088496 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088499 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088502 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088506 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088509 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088512 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088515 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088519 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.089767 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088522 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088525 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088527 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088530 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088533 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088535 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088538 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088541 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088544 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088547 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088549 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088552 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088554 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088557 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088560 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088562 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088565 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088568 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088570 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.088573 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.090239 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089270 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089278 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089282 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089285 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089288 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089292 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089295 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089298 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089301 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089303 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089306 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089308 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089311 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089314 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089316 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089319 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089321 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089324 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089327 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089330 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089333 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.090746 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089335 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089338 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089341 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089343 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089346 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089348 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089351 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089353 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089356 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089359 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089361 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089364 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089366 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089369 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089371 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089374 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089376 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089379 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089382 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089384 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.091268 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089387 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089389 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089392 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089395 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089397 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089400 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089403 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089406 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089409 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089411 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089415 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089418 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089420 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089423 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089425 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089428 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089430 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089433 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089435 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089438 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.091774 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089441 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089443 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089446 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089448 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089465 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089468 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089470 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089473 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089476 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089478 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089482 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089484 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089487 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089490 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089492 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089495 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089499 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089502 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089505 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.092263 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089507 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089510 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089515 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089519 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089521 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089524 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089527 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089529 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089533 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089535 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089538 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089541 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089543 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089546 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089549 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089551 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089554 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089558 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.089561 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.092741 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090028 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090041 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090044 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090048 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090051 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090054 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090057 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090059 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090063 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090065 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090068 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090071 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090073 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090076 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090079 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090081 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090084 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090087 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090090 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090092 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.093228 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090095 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090098 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090101 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090103 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090106 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090109 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090111 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090114 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090116 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090119 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090121 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090124 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090126 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090131 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090134 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090136 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090139 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090141 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090144 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090147 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.093734 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090150 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090152 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090155 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090159 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090163 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090166 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090169 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090172 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090175 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090178 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090180 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090183 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090187 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090192 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090195 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090198 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090200 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090204 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090207 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.094254 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090210 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090212 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090215 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090217 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090220 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090223 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090226 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090230 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090233 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090235 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090239 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090242 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090245 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090248 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090251 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090254 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090256 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090259 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090261 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090264 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.094731 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090266 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090269 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090272 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090274 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090277 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090280 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090282 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090285 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090288 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090292 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090294 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090297 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090299 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090302 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090304 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090307 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090310 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090312 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090315 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.095218 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.090318 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090408 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090451 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090472 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090477 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090482 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090485 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090490 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090494 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090497 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090501 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090504 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090507 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090511 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090514 2559 flags.go:64] FLAG: --cgroup-root="" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090517 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090520 2559 flags.go:64] FLAG: --client-ca-file="" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090524 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090527 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090531 2559 flags.go:64] FLAG: --cluster-domain="" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090534 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090537 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090540 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090543 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 21:15:52.095804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090547 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090550 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090554 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090557 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090560 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090563 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090566 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090569 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090573 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090577 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090581 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090584 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090587 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090590 2559 flags.go:64] FLAG: --enable-server="true" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090593 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090598 2559 flags.go:64] FLAG: --event-burst="100" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090602 2559 flags.go:64] FLAG: --event-qps="50" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090604 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090608 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090611 2559 flags.go:64] FLAG: --eviction-hard="" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090615 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090618 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090622 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090625 2559 flags.go:64] FLAG: --eviction-soft="" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090628 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 21:15:52.096377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090630 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090633 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090637 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090639 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090642 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090645 2559 flags.go:64] FLAG: --feature-gates="" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090649 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090652 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090655 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090658 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090661 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090664 2559 flags.go:64] FLAG: --help="false" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090667 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090670 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090673 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090676 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090679 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090683 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090686 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090689 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090691 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090694 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090697 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090700 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 21:15:52.097023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090703 2559 flags.go:64] FLAG: --kube-reserved="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090707 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090710 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090713 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090716 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090719 2559 flags.go:64] FLAG: --lock-file="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090722 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090725 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090728 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090736 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090739 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090742 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090745 2559 flags.go:64] FLAG: --logging-format="text" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090748 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090751 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090754 2559 flags.go:64] FLAG: --manifest-url="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090757 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090762 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090765 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090769 2559 flags.go:64] FLAG: --max-pods="110" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090772 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090775 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090778 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090781 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090784 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 21:15:52.097636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090786 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090789 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090798 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090801 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090804 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090807 2559 flags.go:64] FLAG: --pod-cidr="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090811 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090816 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090819 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090822 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090825 2559 flags.go:64] FLAG: --port="10250" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090828 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090831 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08ead1de46e1533a7" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090834 2559 flags.go:64] FLAG: --qos-reserved="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090837 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090840 2559 flags.go:64] FLAG: --register-node="true" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090843 2559 flags.go:64] FLAG: --register-with-taints="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090848 2559 flags.go:64] FLAG: --registry-burst="10" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090851 2559 flags.go:64] FLAG: --registry-qps="5" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090854 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090857 2559 flags.go:64] FLAG: --reserved-memory="" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090861 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090863 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090866 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090869 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 21:15:52.098247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090872 2559 flags.go:64] FLAG: --runonce="false" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090875 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090878 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090881 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090884 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090887 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090890 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090893 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090896 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090899 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090902 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090905 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090908 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090910 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090913 2559 flags.go:64] FLAG: --system-cgroups="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090916 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090922 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090925 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090928 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090931 2559 flags.go:64] FLAG: --tls-min-version="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090934 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090937 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090940 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090943 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090948 2559 flags.go:64] FLAG: --v="2" Sep 15 21:15:52.098918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090952 2559 flags.go:64] FLAG: --version="false" Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090957 2559 flags.go:64] FLAG: --vmodule="" Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090960 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.090964 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091079 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091083 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091086 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091089 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091092 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091094 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091097 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091100 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091103 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091105 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091108 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091111 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091114 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091116 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091119 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091122 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.099574 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091125 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091127 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091130 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091132 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091135 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091139 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091142 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091145 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091147 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091150 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091153 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091158 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091160 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091163 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091165 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091168 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091171 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091174 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091176 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091179 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.100072 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091182 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091184 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091187 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091189 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091192 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091194 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091198 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091202 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091205 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091208 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091210 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091213 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091216 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091218 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091221 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091223 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091226 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091230 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091232 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.100572 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091235 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091238 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091240 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091243 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091246 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091249 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091251 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091254 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091257 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091261 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091265 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091269 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091272 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091275 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091278 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091281 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091284 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091287 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091291 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.101045 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091294 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091297 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091300 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091303 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091306 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091308 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091312 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091314 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091316 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091319 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091321 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091326 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091328 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091331 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091333 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091335 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091338 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091342 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091345 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091348 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.101636 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091351 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091353 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091356 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091358 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.091361 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.092291 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.100144 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.100158 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.101062 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.101070 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101127 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101132 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101136 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101139 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101142 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.102274 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101145 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101148 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101152 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101157 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101160 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101163 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101166 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101169 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101171 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101174 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101177 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101179 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101182 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101185 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101188 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101190 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101193 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101195 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101198 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.102686 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101201 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101203 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101206 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101209 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101211 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101215 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101217 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101220 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101223 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101226 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101229 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101231 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101234 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101236 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101239 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101241 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101244 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101247 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101249 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101252 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.103130 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101255 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101257 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101260 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101263 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101266 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101268 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101270 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101273 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101276 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101278 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101281 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101285 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101288 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101291 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101293 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101296 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101298 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101302 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101304 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101307 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.103631 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101309 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101312 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101316 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101319 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101322 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101324 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101327 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101329 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101332 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101335 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101337 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101340 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101342 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101345 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101348 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101350 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101353 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101355 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101358 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101360 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.104117 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101363 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101366 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101368 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101371 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101373 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101376 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101378 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101381 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101383 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101388 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101391 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101395 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101397 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101400 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101403 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.104633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.101408 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101566 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101572 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101575 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101578 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101581 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101584 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101587 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101590 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101593 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101595 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101598 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101600 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101603 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101606 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101608 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101611 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101613 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101616 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101619 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 21:15:52.105003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101621 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101624 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101626 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101629 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101631 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101635 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101638 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101641 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101643 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101647 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101650 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101653 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101655 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101659 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101661 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101664 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101666 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101669 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101671 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 21:15:52.105504 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101674 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101677 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101679 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101682 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101685 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101687 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101690 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101692 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101695 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101697 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101700 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101703 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101707 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101710 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101714 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101717 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101720 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101722 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101726 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 21:15:52.105958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101728 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101731 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101734 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101736 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101739 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101742 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101751 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101754 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101758 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101760 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101763 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101766 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101768 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101771 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101775 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101779 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101781 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101784 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101786 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101789 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 21:15:52.106421 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101791 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101795 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101798 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101801 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101804 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101808 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101810 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101813 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101816 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101818 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101821 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101824 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101826 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101829 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101832 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101834 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101837 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101839 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101842 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 21:15:52.107733 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101845 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 21:15:52.108202 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101847 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 21:15:52.108202 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:52.101850 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 21:15:52.108202 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.101855 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 21:15:52.108202 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.101954 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 21:15:52.108821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.108807 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 21:15:52.109862 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.109850 2559 server.go:1019] "Starting client certificate rotation" Sep 15 21:15:52.109985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.109963 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:15:52.110049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.110011 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 21:15:52.113819 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.113803 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:15:52.119094 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.119073 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 21:15:52.129107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.129091 2559 log.go:25] "Validated CRI v1 runtime API" Sep 15 21:15:52.133033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.133017 2559 log.go:25] "Validated CRI v1 image API" Sep 15 21:15:52.133981 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.133960 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 21:15:52.140620 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.140593 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 954b49c8-0c87-473c-9b3a-be7a6ff37f6b:/dev/nvme0n1p4 c83ccb0f-e621-4674-9878-d73f425be37b:/dev/nvme0n1p3] Sep 15 21:15:52.140702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.140617 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 21:15:52.146195 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.146086 2559 manager.go:216] Machine: {Timestamp:2026-09-15 21:15:52.144574919 +0000 UTC m=+0.369197913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107642 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af0f7620e179d3ec677b9c2191a7f SystemUUID:ec2af0f7-620e-179d-3ec6-77b9c2191a7f BootID:245be705-df8f-4a38-9640-7d09eea39b55 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:15:47:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:15:47:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:14:62:e1:11:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 21:15:52.146195 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.146189 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 21:15:52.146300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.146269 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 21:15:52.146997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.146981 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:15:52.149674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.149649 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 21:15:52.149826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.149676 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 21:15:52.150494 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.150485 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 21:15:52.150526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.150497 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 21:15:52.150526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.150505 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:15:52.151344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.151335 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 21:15:52.152644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.152635 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 21:15:52.157433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157422 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:15:52.157562 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157552 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 21:15:52.157598 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157585 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 15 21:15:52.157598 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157596 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 21:15:52.157655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157611 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 21:15:52.157655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157622 2559 kubelet.go:398] "Adding apiserver pod source" Sep 15 21:15:52.157655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.157630 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 21:15:52.158061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.158048 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:15:52.158113 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.158067 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 21:15:52.161273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.161258 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 21:15:52.164638 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.164625 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 21:15:52.164686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.164656 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 21:15:52.166663 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166651 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166669 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166675 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166681 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166690 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166695 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 21:15:52.166711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166709 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 21:15:52.166859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166716 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 21:15:52.166859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166722 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 21:15:52.166859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166728 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 21:15:52.166859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166741 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 21:15:52.166859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.166747 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 21:15:52.167627 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.167606 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 21:15:52.167664 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.167606 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 21:15:52.167664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.167653 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 21:15:52.167664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.167663 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 21:15:52.171360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.171348 2559 server.go:1268] "Started kubelet" Sep 15 21:15:52.171556 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.171522 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 21:15:52.171644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.171489 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 21:15:52.171807 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.171759 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 21:15:52.176343 ip-10-0-129-100 systemd[1]: Started Kubernetes Kubelet. Sep 15 21:15:52.177832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.177809 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 21:15:52.178809 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.178496 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 15 21:15:52.178809 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.178573 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 21:15:52.184111 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.184094 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 21:15:52.185267 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.184181 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-100.ec2.internal.18d59b177f4376d8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-100.ec2.internal,UID:ip-10-0-129-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-100.ec2.internal,},FirstTimestamp:2026-09-15 21:15:52.171321048 +0000 UTC m=+0.395944043,LastTimestamp:2026-09-15 21:15:52.171321048 +0000 UTC m=+0.395944043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-100.ec2.internal,}" Sep 15 21:15:52.188093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.188078 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 21:15:52.188769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.188754 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 21:15:52.188890 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.188874 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.189499 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.189478 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 21:15:52.189499 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.189484 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 21:15:52.189626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.189506 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 21:15:52.189626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.189581 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 15 21:15:52.189626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.189588 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 15 21:15:52.190353 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.190321 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 21:15:52.190890 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.190623 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 21:15:52.192372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192354 2559 factory.go:55] Registering systemd factory Sep 15 21:15:52.192372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192370 2559 factory.go:223] Registration of the systemd container factory successfully Sep 15 21:15:52.192637 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192622 2559 factory.go:153] Registering CRI-O factory Sep 15 21:15:52.192708 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192641 2559 factory.go:223] Registration of the crio container factory successfully Sep 15 21:15:52.192708 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192696 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 21:15:52.192766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192725 2559 factory.go:103] Registering Raw factory Sep 15 21:15:52.192766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.192741 2559 manager.go:1196] Started watching for new ooms in manager Sep 15 21:15:52.193251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.193239 2559 manager.go:349] Starting recovery of all containers Sep 15 21:15:52.196871 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.196830 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 21:15:52.205319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.205305 2559 manager.go:354] Recovery completed Sep 15 21:15:52.209579 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.209567 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.211430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.211410 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.211565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.211444 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.211565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.211491 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.212192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212176 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 21:15:52.212192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212190 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 21:15:52.212304 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212210 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.212304 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212212 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 15 21:15:52.212979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212962 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.213049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.212992 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.213049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.213014 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.214395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.214381 2559 policy_none.go:49] "None policy: Start" Sep 15 21:15:52.214493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.214399 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 21:15:52.214493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.214409 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 21:15:52.214567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.214510 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnq8w" Sep 15 21:15:52.216490 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.216475 2559 policy_none.go:47] "Start" Sep 15 21:15:52.221012 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.220986 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnq8w" Sep 15 21:15:52.222514 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.222391 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-100.ec2.internal.18d59b1781a77bd4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-100.ec2.internal,UID:ip-10-0-129-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-100.ec2.internal,},FirstTimestamp:2026-09-15 21:15:52.211430356 +0000 UTC m=+0.436053350,LastTimestamp:2026-09-15 21:15:52.211430356 +0000 UTC m=+0.436053350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-100.ec2.internal,}" Sep 15 21:15:52.250151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250131 2559 manager.go:337] "Starting Device Plugin manager" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.250180 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250195 2559 server.go:87] "Starting device plugin registration server" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250493 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250508 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250635 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250717 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.250727 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.251315 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 21:15:52.265998 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.251368 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.283600 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.283574 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 21:15:52.283716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.283635 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 21:15:52.283716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.283662 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 21:15:52.283863 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.283705 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 21:15:52.285589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.285570 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:52.351627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.351563 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.352634 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.352617 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.352745 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.352653 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.352745 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.352668 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.352745 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.352698 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.360663 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.360644 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.360759 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.360671 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-100.ec2.internal\": node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.384322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.384301 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal"] Sep 15 21:15:52.384422 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.384392 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.385773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.385757 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.385847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.385785 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.385847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.385795 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.388049 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.388032 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.388204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388192 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.388324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388309 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.388363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388342 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.388860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388842 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.388962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388870 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.388962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388882 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.388962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388909 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.388962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388929 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.388962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.388939 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.390802 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.390786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.390888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.390814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.390888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.390852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39d7ad530ef8927e7726fb19029c9345-config\") pod \"kube-apiserver-proxy-ip-10-0-129-100.ec2.internal\" (UID: \"39d7ad530ef8927e7726fb19029c9345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.391192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.391178 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.391226 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.391204 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:52.391918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.391902 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:52.391998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.391933 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:52.391998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.391945 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:52.416020 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.416005 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-100.ec2.internal\" not found" node="ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.420273 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.420258 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-100.ec2.internal\" not found" node="ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.488565 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.488377 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.491687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.491822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491691 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.491822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39d7ad530ef8927e7726fb19029c9345-config\") pod \"kube-apiserver-proxy-ip-10-0-129-100.ec2.internal\" (UID: \"39d7ad530ef8927e7726fb19029c9345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.491822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.491822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39d7ad530ef8927e7726fb19029c9345-config\") pod \"kube-apiserver-proxy-ip-10-0-129-100.ec2.internal\" (UID: \"39d7ad530ef8927e7726fb19029c9345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.491958 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.491835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8c06af6dd33d8a03dd3e75bceae61bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal\" (UID: \"c8c06af6dd33d8a03dd3e75bceae61bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.589408 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.589364 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.690204 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.690138 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.718450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.718431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.723811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:52.723400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:52.790230 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.790190 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.892609 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.892580 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:52.992836 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:52.992758 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.093854 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.093827 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.110195 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.110179 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 21:15:53.110319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.110303 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 21:15:53.194572 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.194552 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.227562 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.227525 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:10:52 +0000 UTC" deadline="2028-04-18 02:50:35.077229258 +0000 UTC" Sep 15 21:15:53.227562 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.227558 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13925h34m41.849674622s" Sep 15 21:15:53.295645 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.295581 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.341906 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:53.341853 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d7ad530ef8927e7726fb19029c9345.slice/crio-729f4dacfaa9cddac3b1179d59d8896b27905f5e010f9bff9fe4c02ed988825f WatchSource:0}: Error finding container 729f4dacfaa9cddac3b1179d59d8896b27905f5e010f9bff9fe4c02ed988825f: Status 404 returned error can't find the container with id 729f4dacfaa9cddac3b1179d59d8896b27905f5e010f9bff9fe4c02ed988825f Sep 15 21:15:53.345891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.345877 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:15:53.396286 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.396249 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.404013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.403993 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 21:15:53.404891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.404875 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientMemory" Sep 15 21:15:53.404956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.404906 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 21:15:53.404956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.404920 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeHasSufficientPID" Sep 15 21:15:53.497430 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.497407 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.551294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.551247 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:53.598183 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:53.598153 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-100.ec2.internal\" not found" Sep 15 21:15:53.666026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.666005 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:53.672168 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.672150 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:53.689738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.689717 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" Sep 15 21:15:53.705932 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.705911 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:15:53.706764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.706751 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" Sep 15 21:15:53.717171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:53.717155 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 21:15:54.159763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.159732 2559 apiserver.go:52] "Watching apiserver" Sep 15 21:15:54.170363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.170316 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 21:15:54.172468 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.172115 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ss58r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal","openshift-network-operator/iptables-alerter-x6c2w","openshift-ovn-kubernetes/ovnkube-node-lkd82","kube-system/global-pull-secret-syncer-nckfc","kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d","openshift-image-registry/node-ca-bzm7t","openshift-multus/multus-6lq65","openshift-multus/multus-additional-cni-plugins-lkctb","openshift-multus/network-metrics-daemon-wbzv9","openshift-network-diagnostics/network-check-target-d6fj4","kube-system/konnectivity-agent-45z4b","openshift-cluster-node-tuning-operator/tuned-jktqg"] Sep 15 21:15:54.174990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.174959 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.177662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.177637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.177767 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.177741 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:15:54.178735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.178550 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbrgb\"" Sep 15 21:15:54.178735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.178589 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 21:15:54.178908 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.178889 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.178955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.178940 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.179110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.179094 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 21:15:54.179194 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.179180 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 21:15:54.180136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.180111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.182721 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.182697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.183619 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.183597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 21:15:54.183717 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.183687 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhdv7\"" Sep 15 21:15:54.183717 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.183707 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.183915 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.183895 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.185103 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.185084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.185710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.185452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:54.185710 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.185513 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:15:54.185864 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.185843 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lptwv\"" Sep 15 21:15:54.186099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.186080 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 21:15:54.186313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.186296 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.187722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.187701 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.189722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.189703 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 21:15:54.190462 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.190368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.190567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.190480 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.190634 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.190566 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.190687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.190480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8vzmv\"" Sep 15 21:15:54.191151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.191026 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 21:15:54.192878 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.192855 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 21:15:54.193355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.193338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hfzd\"" Sep 15 21:15:54.193418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.193377 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.195985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.195963 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.196366 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.196345 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 21:15:54.196841 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.196825 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 21:15:54.197014 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.196995 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2h9vl\"" Sep 15 21:15:54.198681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.198664 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.198848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.198828 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.198953 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.198932 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:15:54.199042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.198831 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-czth7\"" Sep 15 21:15:54.199680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.199660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.201299 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-sys-fs\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.201411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mwm\" (UniqueName: \"kubernetes.io/projected/6609b36d-5444-4a82-8e7c-fed235462d3e-kube-api-access-82mwm\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.201411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.201411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:54.201411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-k8s-cni-cncf-io\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201422 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44e40931-3727-41f0-994c-e8adf49cc5ae-agent-certs\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.201645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-socket-dir-parent\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-multus\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-hostroot\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201622 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-conf-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzrt\" (UniqueName: \"kubernetes.io/projected/6d8331bb-178d-429c-96b4-5009a37963b4-kube-api-access-xrzrt\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201691 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4b0234d-2913-4587-b2de-6e9d61fa7293-serviceca\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.201887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201871 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cnibin\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.201995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cf2783e-9b8e-4665-b830-62ceb3892b7b-host-slash\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-bin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44e40931-3727-41f0-994c-e8adf49cc5ae-konnectivity-ca\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.202176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202181 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cf2783e-9b8e-4665-b830-62ceb3892b7b-iptables-alerter-script\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-cnibin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-netns\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-multus-certs\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202315 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-registration-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blzw\" (UniqueName: \"kubernetes.io/projected/b4b0234d-2913-4587-b2de-6e9d61fa7293-kube-api-access-9blzw\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202379 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-system-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-cni-binary-copy\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-etc-kubernetes\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.202609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4b0234d-2913-4587-b2de-6e9d61fa7293-host\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-socket-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-device-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxzg\" (UniqueName: \"kubernetes.io/projected/3e6c6be5-534c-4a3d-b29c-bd667defba9d-kube-api-access-kwxzg\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gnl\" (UniqueName: \"kubernetes.io/projected/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-kube-api-access-r4gnl\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-kubelet\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-multus-daemon-config\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.203162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.202951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.203886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.203013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqvr\" (UniqueName: \"kubernetes.io/projected/0cf2783e-9b8e-4665-b830-62ceb3892b7b-kube-api-access-lqqvr\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.203990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.203917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.204349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.204329 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.206186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.206158 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.206421 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.206400 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.206627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.206605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 21:15:54.206732 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.206711 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 21:15:54.206830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.206813 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 21:15:54.207087 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.207071 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnkn6\"" Sep 15 21:15:54.207742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.207715 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 21:15:54.207867 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.207849 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 21:15:54.208036 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.207856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:15:54.209750 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.208606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6zsv\"" Sep 15 21:15:54.244069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.244045 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-545vx" Sep 15 21:15:54.253675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.253648 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-545vx" Sep 15 21:15:54.289298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.289238 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" event={"ID":"39d7ad530ef8927e7726fb19029c9345","Type":"ContainerStarted","Data":"729f4dacfaa9cddac3b1179d59d8896b27905f5e010f9bff9fe4c02ed988825f"} Sep 15 21:15:54.291270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.291247 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 21:15:54.304703 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-multus-daemon-config\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.304830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304719 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-kubelet\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.304830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-netns\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.304830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.304997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.304997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-tmp-dir\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.304997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mwm\" (UniqueName: \"kubernetes.io/projected/6609b36d-5444-4a82-8e7c-fed235462d3e-kube-api-access-82mwm\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.304997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-k8s-cni-cncf-io\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.304997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.304982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44e40931-3727-41f0-994c-e8adf49cc5ae-agent-certs\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-socket-dir-parent\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-k8s-cni-cncf-io\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305136 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-hostroot\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-conf-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-socket-dir-parent\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305204 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzrt\" (UniqueName: \"kubernetes.io/projected/6d8331bb-178d-429c-96b4-5009a37963b4-kube-api-access-xrzrt\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.305253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-hostroot\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-log-socket\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-multus-conf-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cnibin\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-bin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-multus-daemon-config\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305473 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cnibin\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-etc-selinux\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305556 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-bin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-env-overrides\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgd4k\" (UniqueName: \"kubernetes.io/projected/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-kube-api-access-wgd4k\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305613 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-conf\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305635 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-sys\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cf2783e-9b8e-4665-b830-62ceb3892b7b-iptables-alerter-script\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.305737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-netns\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-node-log\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-netns\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgq9v\" (UniqueName: \"kubernetes.io/projected/a893b465-65a2-436e-b23c-f8f721e35442-kube-api-access-rgq9v\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blzw\" (UniqueName: \"kubernetes.io/projected/b4b0234d-2913-4587-b2de-6e9d61fa7293-kube-api-access-9blzw\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-ovn\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-netd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksn2v\" (UniqueName: \"kubernetes.io/projected/52ed3652-4395-414b-aab7-6ef206895acb-kube-api-access-ksn2v\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-etc-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-run\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.305987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-var-lib-kubelet\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306014 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-etc-tuned\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxzg\" (UniqueName: \"kubernetes.io/projected/3e6c6be5-534c-4a3d-b29c-bd667defba9d-kube-api-access-kwxzg\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.306614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gnl\" (UniqueName: \"kubernetes.io/projected/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-kube-api-access-r4gnl\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52ed3652-4395-414b-aab7-6ef206895acb-ovn-node-metrics-cert\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-lib-modules\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0cf2783e-9b8e-4665-b830-62ceb3892b7b-iptables-alerter-script\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqvr\" (UniqueName: \"kubernetes.io/projected/0cf2783e-9b8e-4665-b830-62ceb3892b7b-kube-api-access-lqqvr\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-slash\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306274 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-bin\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-kubernetes\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-sys-fs\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-var-lib-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-sys-fs\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306547 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-config\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.307301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-script-lib\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-hosts-file\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-modprobe-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-multus\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-kubelet-config\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-cni-multus\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4b0234d-2913-4587-b2de-6e9d61fa7293-serviceca\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306903 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cf2783e-9b8e-4665-b830-62ceb3892b7b-host-slash\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-multus-certs\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.306990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0cf2783e-9b8e-4665-b830-62ceb3892b7b-host-slash\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44e40931-3727-41f0-994c-e8adf49cc5ae-konnectivity-ca\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.307143 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.308035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-cnibin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.307213 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:54.807191915 +0000 UTC m=+3.031814906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307253 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-cnibin\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-run-multus-certs\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307410 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysconfig\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-host\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4b0234d-2913-4587-b2de-6e9d61fa7293-serviceca\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e6c6be5-534c-4a3d-b29c-bd667defba9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-tmp\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e6c6be5-534c-4a3d-b29c-bd667defba9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-registration-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-system-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-registration-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-cni-binary-copy\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-etc-kubernetes\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-system-cni-dir\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307657 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-systemd-units\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.308896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-systemd\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307686 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-etc-kubernetes\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4b0234d-2913-4587-b2de-6e9d61fa7293-host\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-systemd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4b0234d-2913-4587-b2de-6e9d61fa7293-host\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44e40931-3727-41f0-994c-e8adf49cc5ae-konnectivity-ca\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-dbus\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-socket-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-device-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-kubelet\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-device-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.307988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6609b36d-5444-4a82-8e7c-fed235462d3e-socket-dir\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.308000 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8331bb-178d-429c-96b4-5009a37963b4-host-var-lib-kubelet\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.309741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.308130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d8331bb-178d-429c-96b4-5009a37963b4-cni-binary-copy\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.311001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.310982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44e40931-3727-41f0-994c-e8adf49cc5ae-agent-certs\") pod \"konnectivity-agent-45z4b\" (UID: \"44e40931-3727-41f0-994c-e8adf49cc5ae\") " pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.314052 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.313878 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:15:54.314052 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.313954 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:15:54.314052 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.313972 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:54.314052 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.314032 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:15:54.814014304 +0000 UTC m=+3.038637288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:54.315850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.315827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqvr\" (UniqueName: \"kubernetes.io/projected/0cf2783e-9b8e-4665-b830-62ceb3892b7b-kube-api-access-lqqvr\") pod \"iptables-alerter-x6c2w\" (UID: \"0cf2783e-9b8e-4665-b830-62ceb3892b7b\") " pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.316008 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.315982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxzg\" (UniqueName: \"kubernetes.io/projected/3e6c6be5-534c-4a3d-b29c-bd667defba9d-kube-api-access-kwxzg\") pod \"multus-additional-cni-plugins-lkctb\" (UID: \"3e6c6be5-534c-4a3d-b29c-bd667defba9d\") " pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.316223 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.316196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mwm\" (UniqueName: \"kubernetes.io/projected/6609b36d-5444-4a82-8e7c-fed235462d3e-kube-api-access-82mwm\") pod \"aws-ebs-csi-driver-node-ssx4d\" (UID: \"6609b36d-5444-4a82-8e7c-fed235462d3e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.318250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.318202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzrt\" (UniqueName: \"kubernetes.io/projected/6d8331bb-178d-429c-96b4-5009a37963b4-kube-api-access-xrzrt\") pod \"multus-6lq65\" (UID: \"6d8331bb-178d-429c-96b4-5009a37963b4\") " pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.319332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.319311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blzw\" (UniqueName: \"kubernetes.io/projected/b4b0234d-2913-4587-b2de-6e9d61fa7293-kube-api-access-9blzw\") pod \"node-ca-bzm7t\" (UID: \"b4b0234d-2913-4587-b2de-6e9d61fa7293\") " pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.319626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.319601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gnl\" (UniqueName: \"kubernetes.io/projected/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-kube-api-access-r4gnl\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.408192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-log-socket\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-env-overrides\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgd4k\" (UniqueName: \"kubernetes.io/projected/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-kube-api-access-wgd4k\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-log-socket\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-conf\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-sys\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-node-log\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgq9v\" (UniqueName: \"kubernetes.io/projected/a893b465-65a2-436e-b23c-f8f721e35442-kube-api-access-rgq9v\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-conf\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-ovn\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408455 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-sys\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-netd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-ovn\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-node-log\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksn2v\" (UniqueName: \"kubernetes.io/projected/52ed3652-4395-414b-aab7-6ef206895acb-kube-api-access-ksn2v\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-etc-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-netd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-run\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-var-lib-kubelet\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-etc-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-run\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-etc-tuned\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.408770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52ed3652-4395-414b-aab7-6ef206895acb-ovn-node-metrics-cert\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408686 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-var-lib-kubelet\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-lib-modules\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-slash\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-bin\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-kubernetes\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-slash\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408818 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-var-lib-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-env-overrides\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-config\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-lib-modules\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-script-lib\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-kubernetes\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-var-lib-openvswitch\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408868 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-cni-bin\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-hosts-file\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-hosts-file\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.408974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-modprobe-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.409615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-kubelet-config\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-kubelet-config\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-modprobe-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysconfig\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409261 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-host\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-tmp\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-systemd-units\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-systemd\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-systemd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-dbus\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-config\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysctl-d\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52ed3652-4395-414b-aab7-6ef206895acb-ovnkube-script-lib\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.410606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-kubelet\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-run-systemd\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-systemd\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-netns\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-host\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-systemd-units\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a893b465-65a2-436e-b23c-f8f721e35442-etc-sysconfig\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-tmp-dir\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-kubelet\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c426a60-e8a8-4486-88f5-af838b4007c0-dbus\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.409603 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ed3652-4395-414b-aab7-6ef206895acb-host-run-netns\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.409672 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:54.909651774 +0000 UTC m=+3.134274773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.409877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-tmp-dir\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.410957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-etc-tuned\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.411498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.411099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52ed3652-4395-414b-aab7-6ef206895acb-ovn-node-metrics-cert\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.412084 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.411792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a893b465-65a2-436e-b23c-f8f721e35442-tmp\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.423042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.423017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgq9v\" (UniqueName: \"kubernetes.io/projected/a893b465-65a2-436e-b23c-f8f721e35442-kube-api-access-rgq9v\") pod \"tuned-jktqg\" (UID: \"a893b465-65a2-436e-b23c-f8f721e35442\") " pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.424186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.424166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgd4k\" (UniqueName: \"kubernetes.io/projected/947e6bd7-0aef-424b-8c05-ad8ac9aa28e7-kube-api-access-wgd4k\") pod \"node-resolver-ss58r\" (UID: \"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7\") " pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.425587 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.425567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksn2v\" (UniqueName: \"kubernetes.io/projected/52ed3652-4395-414b-aab7-6ef206895acb-kube-api-access-ksn2v\") pod \"ovnkube-node-lkd82\" (UID: \"52ed3652-4395-414b-aab7-6ef206895acb\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.491300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.491275 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkctb" Sep 15 21:15:54.497497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.497469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6c2w" Sep 15 21:15:54.509285 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.509261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzm7t" Sep 15 21:15:54.520430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.520408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" Sep 15 21:15:54.527305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.527286 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6lq65" Sep 15 21:15:54.536055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.536031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:15:54.543337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.543306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss58r" Sep 15 21:15:54.552218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.552197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:15:54.559045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.559024 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jktqg" Sep 15 21:15:54.664899 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.664820 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 21:15:54.812654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.812615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:54.812807 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.812775 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:54.812850 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.812841 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:55.812826074 +0000 UTC m=+4.037449072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:54.913411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.913368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:54.913604 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:54.913553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:54.913604 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913579 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:15:54.913604 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913604 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:15:54.913768 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913619 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:54.913768 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913695 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:15:55.913672 +0000 UTC m=+4.138295004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:54.913768 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913698 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:54.913768 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:54.913754 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:55.913738435 +0000 UTC m=+4.138361435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:55.105055 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.105015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947e6bd7_0aef_424b_8c05_ad8ac9aa28e7.slice/crio-9503b1296e13bcfa792d5427bafad5888a3a9dedbd0174a7bfd1af4a5008cda3 WatchSource:0}: Error finding container 9503b1296e13bcfa792d5427bafad5888a3a9dedbd0174a7bfd1af4a5008cda3: Status 404 returned error can't find the container with id 9503b1296e13bcfa792d5427bafad5888a3a9dedbd0174a7bfd1af4a5008cda3 Sep 15 21:15:55.106161 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.106119 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf2783e_9b8e_4665_b830_62ceb3892b7b.slice/crio-e61ccc51598f91876da7c0d6c4c00ece4c36d0087d8184b51222ab16abab70f6 WatchSource:0}: Error finding container e61ccc51598f91876da7c0d6c4c00ece4c36d0087d8184b51222ab16abab70f6: Status 404 returned error can't find the container with id e61ccc51598f91876da7c0d6c4c00ece4c36d0087d8184b51222ab16abab70f6 Sep 15 21:15:55.110163 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.110124 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ed3652_4395_414b_aab7_6ef206895acb.slice/crio-b3d602bae61225c86d56d492439f4e7ee13d5288278ca7f6c7d5e641552cd86c WatchSource:0}: Error finding container b3d602bae61225c86d56d492439f4e7ee13d5288278ca7f6c7d5e641552cd86c: Status 404 returned error can't find the container with id b3d602bae61225c86d56d492439f4e7ee13d5288278ca7f6c7d5e641552cd86c Sep 15 21:15:55.111033 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.110995 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b0234d_2913_4587_b2de_6e9d61fa7293.slice/crio-e2be29630c39ad49fb9e8a4572d13b6680bd7e0a758385ee44063250a9be90fd WatchSource:0}: Error finding container e2be29630c39ad49fb9e8a4572d13b6680bd7e0a758385ee44063250a9be90fd: Status 404 returned error can't find the container with id e2be29630c39ad49fb9e8a4572d13b6680bd7e0a758385ee44063250a9be90fd Sep 15 21:15:55.111550 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.111525 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e40931_3727_41f0_994c_e8adf49cc5ae.slice/crio-afb5b9c9d4d4ab250b8f6bf31c514d7fd463f61a4354a89db916834bc82fdf0e WatchSource:0}: Error finding container afb5b9c9d4d4ab250b8f6bf31c514d7fd463f61a4354a89db916834bc82fdf0e: Status 404 returned error can't find the container with id afb5b9c9d4d4ab250b8f6bf31c514d7fd463f61a4354a89db916834bc82fdf0e Sep 15 21:15:55.114166 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.113588 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8331bb_178d_429c_96b4_5009a37963b4.slice/crio-a256f92bf8b0c7ae9af22057ff5a34e17a633fa28f49119270f2d9b8d1417a1f WatchSource:0}: Error finding container a256f92bf8b0c7ae9af22057ff5a34e17a633fa28f49119270f2d9b8d1417a1f: Status 404 returned error can't find the container with id a256f92bf8b0c7ae9af22057ff5a34e17a633fa28f49119270f2d9b8d1417a1f Sep 15 21:15:55.114166 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.114016 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda893b465_65a2_436e_b23c_f8f721e35442.slice/crio-a98fba7bdb7206ab3a3d785e6d4a26f3309cba0be8266cf0a9912d856d2b349f WatchSource:0}: Error finding container a98fba7bdb7206ab3a3d785e6d4a26f3309cba0be8266cf0a9912d856d2b349f: Status 404 returned error can't find the container with id a98fba7bdb7206ab3a3d785e6d4a26f3309cba0be8266cf0a9912d856d2b349f Sep 15 21:15:55.115717 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.115651 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6c6be5_534c_4a3d_b29c_bd667defba9d.slice/crio-f544029850a3cd9ffe6ea28c53848c785780271057f06637152254d717f8b162 WatchSource:0}: Error finding container f544029850a3cd9ffe6ea28c53848c785780271057f06637152254d717f8b162: Status 404 returned error can't find the container with id f544029850a3cd9ffe6ea28c53848c785780271057f06637152254d717f8b162 Sep 15 21:15:55.116418 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.116392 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6609b36d_5444_4a82_8e7c_fed235462d3e.slice/crio-48aa9791fe4cef4fca8ee807c43418dfa6b97699613babcdab380c048e2ed7cc WatchSource:0}: Error finding container 48aa9791fe4cef4fca8ee807c43418dfa6b97699613babcdab380c048e2ed7cc: Status 404 returned error can't find the container with id 48aa9791fe4cef4fca8ee807c43418dfa6b97699613babcdab380c048e2ed7cc Sep 15 21:15:55.163203 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:15:55.163173 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c06af6dd33d8a03dd3e75bceae61bb.slice/crio-ee09e9626137a46c820e29999e05e303cbd9aefee7efeb45d38757c63f1dfabf WatchSource:0}: Error finding container ee09e9626137a46c820e29999e05e303cbd9aefee7efeb45d38757c63f1dfabf: Status 404 returned error can't find the container with id ee09e9626137a46c820e29999e05e303cbd9aefee7efeb45d38757c63f1dfabf Sep 15 21:15:55.257764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.257721 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:10:54 +0000 UTC" deadline="2028-03-19 02:13:06.544692523 +0000 UTC" Sep 15 21:15:55.257764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.257760 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13204h57m11.286935357s" Sep 15 21:15:55.284260 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.284229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:55.284372 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.284352 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:15:55.293441 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.293410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" event={"ID":"39d7ad530ef8927e7726fb19029c9345","Type":"ContainerStarted","Data":"fd0ce42574428a1d841500e74e2e0ffdbf1d800e5842f8d872c919c6c97d72ae"} Sep 15 21:15:55.294611 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.294590 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" event={"ID":"6609b36d-5444-4a82-8e7c-fed235462d3e","Type":"ContainerStarted","Data":"48aa9791fe4cef4fca8ee807c43418dfa6b97699613babcdab380c048e2ed7cc"} Sep 15 21:15:55.295759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.295734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6lq65" event={"ID":"6d8331bb-178d-429c-96b4-5009a37963b4","Type":"ContainerStarted","Data":"a256f92bf8b0c7ae9af22057ff5a34e17a633fa28f49119270f2d9b8d1417a1f"} Sep 15 21:15:55.296987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.296968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"b3d602bae61225c86d56d492439f4e7ee13d5288278ca7f6c7d5e641552cd86c"} Sep 15 21:15:55.298253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.298227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6c2w" event={"ID":"0cf2783e-9b8e-4665-b830-62ceb3892b7b","Type":"ContainerStarted","Data":"e61ccc51598f91876da7c0d6c4c00ece4c36d0087d8184b51222ab16abab70f6"} Sep 15 21:15:55.299401 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.299378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" event={"ID":"c8c06af6dd33d8a03dd3e75bceae61bb","Type":"ContainerStarted","Data":"ee09e9626137a46c820e29999e05e303cbd9aefee7efeb45d38757c63f1dfabf"} Sep 15 21:15:55.300526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.300505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerStarted","Data":"f544029850a3cd9ffe6ea28c53848c785780271057f06637152254d717f8b162"} Sep 15 21:15:55.301590 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.301559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jktqg" event={"ID":"a893b465-65a2-436e-b23c-f8f721e35442","Type":"ContainerStarted","Data":"a98fba7bdb7206ab3a3d785e6d4a26f3309cba0be8266cf0a9912d856d2b349f"} Sep 15 21:15:55.302600 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.302582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45z4b" event={"ID":"44e40931-3727-41f0-994c-e8adf49cc5ae","Type":"ContainerStarted","Data":"afb5b9c9d4d4ab250b8f6bf31c514d7fd463f61a4354a89db916834bc82fdf0e"} Sep 15 21:15:55.303647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.303627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzm7t" event={"ID":"b4b0234d-2913-4587-b2de-6e9d61fa7293","Type":"ContainerStarted","Data":"e2be29630c39ad49fb9e8a4572d13b6680bd7e0a758385ee44063250a9be90fd"} Sep 15 21:15:55.304644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.304626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss58r" event={"ID":"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7","Type":"ContainerStarted","Data":"9503b1296e13bcfa792d5427bafad5888a3a9dedbd0174a7bfd1af4a5008cda3"} Sep 15 21:15:55.314414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.314357 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-100.ec2.internal" podStartSLOduration=2.314344917 podStartE2EDuration="2.314344917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:55.313964586 +0000 UTC m=+3.538587594" watchObservedRunningTime="2026-09-15 21:15:55.314344917 +0000 UTC m=+3.538967924" Sep 15 21:15:55.822944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.822900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:55.823133 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.823056 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:55.823133 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.823123 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:57.823102698 +0000 UTC m=+6.047725685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.923253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:55.923338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.923478 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.923562 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:15:57.923541559 +0000 UTC m=+6.148164566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.923975 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.923996 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.924009 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:55.924104 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:55.924052 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:15:57.924037978 +0000 UTC m=+6.148660963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:56.261289 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:56.261184 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:10:54 +0000 UTC" deadline="2028-06-03 22:43:08.62348179 +0000 UTC" Sep 15 21:15:56.261289 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:56.261240 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15049h27m12.362245824s" Sep 15 21:15:56.287860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:56.287116 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:56.287860 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:56.287240 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:15:56.287860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:56.287674 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:56.287860 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:56.287774 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:15:57.284722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:57.284121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:57.284722 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.284287 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:15:57.840286 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:57.840253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:57.840476 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.840408 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:57.840561 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.840515 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:01.84049422 +0000 UTC m=+10.065117226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:15:57.940675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:57.940633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:57.940859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:57.940728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:57.940928 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.940881 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:57.940977 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.940944 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:01.940927282 +0000 UTC m=+10.165550285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:15:57.941381 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.941361 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:15:57.941489 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.941390 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:15:57.941489 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.941414 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:57.941489 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:57.941479 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:16:01.941448461 +0000 UTC m=+10.166071446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:15:58.286794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:58.286031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:15:58.286794 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:58.286156 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:15:58.286794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:58.286552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:15:58.286794 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:58.286642 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:15:59.284938 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:15:59.284446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:15:59.284938 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:15:59.284595 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:00.284636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:00.284596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:00.285075 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:00.284742 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:00.285147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:00.285117 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:00.286414 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:00.285319 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:01.284542 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:01.284272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:01.284717 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.284691 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:01.877669 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:01.877636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:01.877834 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.877791 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:01.877915 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.877854 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.877835225 +0000 UTC m=+18.102458227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:01.978379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:01.978340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:01.978559 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:01.978434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:01.978559 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978516 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:01.978559 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978541 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:01.978559 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978553 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:01.978752 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978565 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:01.978752 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978611 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.978593203 +0000 UTC m=+18.203216186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:01.978752 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:01.978631 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:09.978621749 +0000 UTC m=+18.203244739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:02.285448 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:02.285338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:02.285883 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:02.285449 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:02.285883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:02.285811 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:02.286000 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:02.285898 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:03.284635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:03.284581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:03.284801 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:03.284730 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:04.284015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:04.283975 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:04.284386 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:04.284096 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:04.284386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:04.283988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:04.284566 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:04.284522 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:05.284778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:05.284746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:05.285215 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:05.284879 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:06.283854 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:06.283828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:06.283970 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:06.283941 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:06.284039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:06.284013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:06.284180 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:06.284155 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:07.284679 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:07.284649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:07.285067 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:07.284790 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:08.284238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:08.284204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:08.284400 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:08.284332 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:08.284493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:08.284410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:08.284547 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:08.284523 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:09.284377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:09.284349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:09.284830 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:09.284447 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:09.936017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:09.935978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:09.936185 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:09.936161 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:09.936252 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:09.936238 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:25.93621701 +0000 UTC m=+34.160840005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:10.037290 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:10.037251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:10.037496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:10.037331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:10.037496 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037447 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:10.037602 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037526 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:26.037512788 +0000 UTC m=+34.262135784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:10.037602 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037467 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:10.037695 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037599 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:10.037695 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037627 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:10.037695 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.037681 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:16:26.037663773 +0000 UTC m=+34.262286774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:10.284355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:10.284278 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:10.284528 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.284400 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:10.284528 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:10.284464 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:10.284938 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:10.284565 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:11.284609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:11.284577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:11.285081 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:11.284706 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:12.284887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:12.284840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:12.285308 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:12.285006 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:12.285308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:12.285016 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:12.285308 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:12.285105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:13.283947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.283922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:13.284084 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:13.284057 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:13.365567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.365499 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6lq65" event={"ID":"6d8331bb-178d-429c-96b4-5009a37963b4","Type":"ContainerStarted","Data":"7c47a82f995a4509cd46293328dea70c61e83c8e90e24ce5fa37bb27875dbd3d"} Sep 15 21:16:13.381597 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.381348 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"25c5805d8c064060072bd6f7cd1c610e71add86813ab914f21dc7fefe4542b56"} Sep 15 21:16:13.385445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.385414 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jktqg" event={"ID":"a893b465-65a2-436e-b23c-f8f721e35442","Type":"ContainerStarted","Data":"783b48ddf5cb7a10e806c78520156727e2909c1e3cf7de5b6470e4e76b59bac2"} Sep 15 21:16:13.391836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.391749 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6lq65" podStartSLOduration=3.320542783 podStartE2EDuration="21.391729416s" totalImagesPullingTime="18.071186633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:13.391311265 +0000 UTC m=+21.615934265" watchObservedRunningTime="2026-09-15 21:16:13.391729416 +0000 UTC m=+21.616352423" Sep 15 21:16:13.413117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:13.413064 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jktqg" podStartSLOduration=2.381623104 podStartE2EDuration="20.413045694s" totalImagesPullingTime="18.03142259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:13.412954439 +0000 UTC m=+21.637577447" watchObservedRunningTime="2026-09-15 21:16:13.413045694 +0000 UTC m=+21.637668699" Sep 15 21:16:14.284713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.284684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:14.284888 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:14.284815 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:14.284965 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.284893 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:14.285020 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:14.284979 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:14.389564 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.389530 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" event={"ID":"6609b36d-5444-4a82-8e7c-fed235462d3e","Type":"ContainerStarted","Data":"5491bd277d179c8091fe5ef3dabc6ada0f5e684e4b0478e2bba36d89dbbfa7a3"} Sep 15 21:16:14.392992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.392965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"5589e7f20bcd494ef8a85b84bb430fcaea086e1bae18867cb98e3514dc844654"} Sep 15 21:16:14.393093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.393032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"4663c30ec4cb35d66d3227ec5370f15ece1f4a5b82ead1dd383a3cd11a016f0c"} Sep 15 21:16:14.393093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.393080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"5db214c008d4c6b73d649919739c33356a087312694b3fed9739c91a933a994e"} Sep 15 21:16:14.393181 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.393133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"5cdeaef21b0ce1f96c7b8d84eaf7284a883dbdf7fc254b6c84e6415c19f513f6"} Sep 15 21:16:14.393214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.393181 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"7c8a513c6467457b4b4a50fc9506146bcd0c0e1ded3fba3b4a59f9178477776e"} Sep 15 21:16:14.396107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.396074 2559 generic.go:356] "Generic (PLEG): container finished" podID="c8c06af6dd33d8a03dd3e75bceae61bb" containerID="166d01130f07458b413ead5aa4be0c9fc9a897da2ce591d0838140267915d7e1" exitCode=0 Sep 15 21:16:14.396217 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.396165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" event={"ID":"c8c06af6dd33d8a03dd3e75bceae61bb","Type":"ContainerDied","Data":"166d01130f07458b413ead5aa4be0c9fc9a897da2ce591d0838140267915d7e1"} Sep 15 21:16:14.397654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.397630 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="46d37c7441b81f3498570d87fd44df1772b89b5fbbc9f6b4f1e59139013e5a35" exitCode=0 Sep 15 21:16:14.397753 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.397707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"46d37c7441b81f3498570d87fd44df1772b89b5fbbc9f6b4f1e59139013e5a35"} Sep 15 21:16:14.399568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.399542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45z4b" event={"ID":"44e40931-3727-41f0-994c-e8adf49cc5ae","Type":"ContainerStarted","Data":"70052c5b74fab6b05940887ce3a5e7bbc3a0292d13bee82c4a9ddf3b8bf9b54c"} Sep 15 21:16:14.401164 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.401140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzm7t" event={"ID":"b4b0234d-2913-4587-b2de-6e9d61fa7293","Type":"ContainerStarted","Data":"ad7749369507b6d604d261a7355903095e49d352dbfbf8656e0b3185c16fdc75"} Sep 15 21:16:14.402557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.402537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss58r" event={"ID":"947e6bd7-0aef-424b-8c05-ad8ac9aa28e7","Type":"ContainerStarted","Data":"be544f42d4a09dc1d65ef0dfe8fa80fb9b8301c5459dd656a038172fa5e38bb9"} Sep 15 21:16:14.465403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.465231 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzm7t" podStartSLOduration=4.420369758 podStartE2EDuration="22.465216016s" totalImagesPullingTime="18.044846258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:14.450293227 +0000 UTC m=+22.674916232" watchObservedRunningTime="2026-09-15 21:16:14.465216016 +0000 UTC m=+22.689839020" Sep 15 21:16:14.465648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.465618 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-45z4b" podStartSLOduration=4.472959313 podStartE2EDuration="22.46560989s" totalImagesPullingTime="17.992650577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:14.464916502 +0000 UTC m=+22.689539507" watchObservedRunningTime="2026-09-15 21:16:14.46560989 +0000 UTC m=+22.690232893" Sep 15 21:16:14.479794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:14.479752 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ss58r" podStartSLOduration=4.420039431 podStartE2EDuration="22.4797384s" totalImagesPullingTime="18.059698969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:14.479691814 +0000 UTC m=+22.704314819" watchObservedRunningTime="2026-09-15 21:16:14.4797384 +0000 UTC m=+22.704361403" Sep 15 21:16:15.128281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.128256 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 21:16:15.271418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.271279 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:16:15.128278634Z","UUID":"4f889995-3cb5-467c-875e-45be273370c4","Handler":null,"Name":"","Endpoint":""} Sep 15 21:16:15.272694 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.272674 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 21:16:15.272804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.272707 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 21:16:15.284468 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.284437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:15.284557 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:15.284539 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:15.407006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.406976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" event={"ID":"6609b36d-5444-4a82-8e7c-fed235462d3e","Type":"ContainerStarted","Data":"a2900c727ea802a5a758b0d753077cc8296d241fb73d227b9ef07d16aac5de7f"} Sep 15 21:16:15.408738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.408706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6c2w" event={"ID":"0cf2783e-9b8e-4665-b830-62ceb3892b7b","Type":"ContainerStarted","Data":"db9d503d9fa09263c29d2df1b28a19a795a6dab0585612cc6a9cb1fd47bd3ec3"} Sep 15 21:16:15.410716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.410689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" event={"ID":"c8c06af6dd33d8a03dd3e75bceae61bb","Type":"ContainerStarted","Data":"233404c51261224b67f1e3296fbfff8f2d95e4fbf7501206cd1f1239745e74be"} Sep 15 21:16:15.426262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:15.426223 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x6c2w" podStartSLOduration=5.373585899 podStartE2EDuration="23.426209481s" totalImagesPullingTime="18.052623582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:15.4259801 +0000 UTC m=+23.650603104" watchObservedRunningTime="2026-09-15 21:16:15.426209481 +0000 UTC m=+23.650832482" Sep 15 21:16:16.284181 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:16.284149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:16.284181 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:16.284173 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:16.284403 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:16.284274 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:16.284403 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:16.284387 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:16.417380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:16.417334 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"3bff4c2687fa988324674dbcdd2244597d62f54e908ba059438883520015d11d"} Sep 15 21:16:17.284647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:17.284621 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:17.284858 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:17.284741 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:17.421529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:17.421492 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" event={"ID":"6609b36d-5444-4a82-8e7c-fed235462d3e","Type":"ContainerStarted","Data":"53ec3287ffd36f5c70fef087ef7264e8b9c2a8416a74149aa922bd68d4fd6943"} Sep 15 21:16:17.440357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:17.440313 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-100.ec2.internal" podStartSLOduration=24.440299381 podStartE2EDuration="24.440299381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:15.441149834 +0000 UTC m=+23.665772838" watchObservedRunningTime="2026-09-15 21:16:17.440299381 +0000 UTC m=+25.664922384" Sep 15 21:16:17.440609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:17.440586 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ssx4d" podStartSLOduration=4.736751769 podStartE2EDuration="25.440580416s" totalImagesPullingTime="20.703828647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:17.439935368 +0000 UTC m=+25.664558374" watchObservedRunningTime="2026-09-15 21:16:17.440580416 +0000 UTC m=+25.665203453" Sep 15 21:16:18.284044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:18.284013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:18.284175 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:18.284013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:18.284175 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:18.284139 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:18.284306 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:18.284204 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:19.284463 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.284426 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:19.284748 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:19.284539 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:19.368883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.368637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:16:19.369388 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.369239 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:16:19.428347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.428322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" event={"ID":"52ed3652-4395-414b-aab7-6ef206895acb","Type":"ContainerStarted","Data":"089c89da9a3d11bcd90b272f249b10020467cea9fd7d94bdf563c90935db8f81"} Sep 15 21:16:19.428634 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.428592 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:19.428708 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.428665 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:19.443357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.443339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:19.460648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:19.460612 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" podStartSLOduration=8.366256071 podStartE2EDuration="26.460599691s" totalImagesPullingTime="18.09434362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:19.459309475 +0000 UTC m=+27.683932478" watchObservedRunningTime="2026-09-15 21:16:19.460599691 +0000 UTC m=+27.685222695" Sep 15 21:16:20.284788 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.284756 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:20.285549 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:20.284861 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:20.285549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.284917 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:20.285549 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:20.285020 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:20.431804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.431766 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="e146e700fd5a627aefecb86cbd7ff5c4cea8f2e903fb9d3447e0c02fd60384c1" exitCode=0 Sep 15 21:16:20.431959 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.431850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"e146e700fd5a627aefecb86cbd7ff5c4cea8f2e903fb9d3447e0c02fd60384c1"} Sep 15 21:16:20.432303 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.432280 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:20.446778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:20.446757 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:21.284360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.284195 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:21.284550 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:21.284528 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:21.299391 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.299365 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:16:21.299804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.299474 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 21:16:21.299848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.299833 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-45z4b" Sep 15 21:16:21.347632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.347554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nckfc"] Sep 15 21:16:21.347719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.347642 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:21.347767 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:21.347713 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:21.351324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.351305 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d6fj4"] Sep 15 21:16:21.351406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.351385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:21.351490 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:21.351468 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:21.352002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.351980 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbzv9"] Sep 15 21:16:21.435308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.435287 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="0713ae06c5f6ce9f37f250aace89d4a6d3cb96aeef1669dcb6dadf2fcb869f32" exitCode=0 Sep 15 21:16:21.435427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.435341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:21.435427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:21.435383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"0713ae06c5f6ce9f37f250aace89d4a6d3cb96aeef1669dcb6dadf2fcb869f32"} Sep 15 21:16:21.435580 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:21.435555 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:23.284021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:23.283943 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:23.284021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:23.283979 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:23.284383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:23.284030 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:23.284383 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:23.284042 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:23.284383 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:23.284097 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:23.284383 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:23.284168 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:24.445200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:24.445168 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="d9ce30968e38a1f2e0543c5e74e5defd88c44e073e221fad57bdbc3b62c9a4bb" exitCode=0 Sep 15 21:16:24.445601 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:24.445209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"d9ce30968e38a1f2e0543c5e74e5defd88c44e073e221fad57bdbc3b62c9a4bb"} Sep 15 21:16:25.283928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:25.283896 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:25.283928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:25.283930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:25.284245 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:25.284045 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbzv9" podUID="b29d6225-c7eb-4ea1-9aa2-e09416b0ff26" Sep 15 21:16:25.284245 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:25.284101 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nckfc" podUID="3c426a60-e8a8-4486-88f5-af838b4007c0" Sep 15 21:16:25.284245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:25.284149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:25.284398 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:25.284265 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6fj4" podUID="7db231b3-6660-4efc-806d-33ce05c8d31a" Sep 15 21:16:25.958225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:25.958194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:25.958599 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:25.958329 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:25.958599 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:25.958383 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs podName:b29d6225-c7eb-4ea1-9aa2-e09416b0ff26 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:57.958368923 +0000 UTC m=+66.182991917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs") pod "network-metrics-daemon-wbzv9" (UID: "b29d6225-c7eb-4ea1-9aa2-e09416b0ff26") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 21:16:26.059403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:26.059369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:26.059590 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:26.059428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:26.059590 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059551 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 21:16:26.059590 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059571 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 21:16:26.059590 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059581 2559 projected.go:196] Error preparing data for projected volume kube-api-access-6xhbb for pod openshift-network-diagnostics/network-check-target-d6fj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:26.059590 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059583 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:26.059851 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059626 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb podName:7db231b3-6660-4efc-806d-33ce05c8d31a nodeName:}" failed. No retries permitted until 2026-09-15 21:16:58.059614021 +0000 UTC m=+66.284237008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhbb" (UniqueName: "kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb") pod "network-check-target-d6fj4" (UID: "7db231b3-6660-4efc-806d-33ce05c8d31a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 21:16:26.059851 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:26.059640 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret podName:3c426a60-e8a8-4486-88f5-af838b4007c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:58.059634046 +0000 UTC m=+66.284257028 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret") pod "global-pull-secret-syncer-nckfc" (UID: "3c426a60-e8a8-4486-88f5-af838b4007c0") : object "kube-system"/"original-pull-secret" not registered Sep 15 21:16:27.114758 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.114725 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-100.ec2.internal" event="NodeReady" Sep 15 21:16:27.115132 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.114864 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 21:16:27.210655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.210624 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:16:27.225936 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.225917 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.231913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.231891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 21:16:27.232748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.232361 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 21:16:27.232748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.232380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 21:16:27.233010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.232984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzsgv\"" Sep 15 21:16:27.234822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.234790 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zdkrl"] Sep 15 21:16:27.253273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.253253 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 21:16:27.254143 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.254127 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mdqkl"] Sep 15 21:16:27.254264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.254247 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.257242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.257226 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 21:16:27.257496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.257481 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ztstg\"" Sep 15 21:16:27.258213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.258194 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 21:16:27.280125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.280104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zdkrl"] Sep 15 21:16:27.280234 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.280142 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:16:27.280234 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.280162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdqkl"] Sep 15 21:16:27.280340 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.280248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.283841 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.283822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:27.283944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.283891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k7wrd\"" Sep 15 21:16:27.283944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.283829 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 21:16:27.283944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.283932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:27.284131 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.283829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:27.284801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.284783 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 21:16:27.285022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.285006 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 21:16:27.288655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288637 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:16:27.288761 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:16:27.288891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288876 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\"" Sep 15 21:16:27.288964 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288948 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 21:16:27.289016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288959 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\"" Sep 15 21:16:27.289016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.288959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 21:16:27.367909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.367840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178eccc4-c353-40e4-aec0-c2aa478cb2fe-config-volume\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.367909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.367881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.367909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.367939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.367965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whg68\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhxf\" (UniqueName: \"kubernetes.io/projected/a541b194-706b-4755-a04d-248fd6007a91-kube-api-access-fmhxf\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368101 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178eccc4-c353-40e4-aec0-c2aa478cb2fe-tmp-dir\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrbn\" (UniqueName: \"kubernetes.io/projected/178eccc4-c353-40e4-aec0-c2aa478cb2fe-kube-api-access-khrbn\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a541b194-706b-4755-a04d-248fd6007a91-cert\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.368314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.368241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.410718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.410672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:27.469113 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469113 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whg68\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhxf\" (UniqueName: \"kubernetes.io/projected/a541b194-706b-4755-a04d-248fd6007a91-kube-api-access-fmhxf\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.469288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178eccc4-c353-40e4-aec0-c2aa478cb2fe-tmp-dir\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.469288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrbn\" (UniqueName: \"kubernetes.io/projected/178eccc4-c353-40e4-aec0-c2aa478cb2fe-kube-api-access-khrbn\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.469288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a541b194-706b-4755-a04d-248fd6007a91-cert\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.469519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.469519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178eccc4-c353-40e4-aec0-c2aa478cb2fe-config-volume\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.469653 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.469542 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:27.469653 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.469610 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:27.969583182 +0000 UTC m=+36.194206180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:27.469836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178eccc4-c353-40e4-aec0-c2aa478cb2fe-tmp-dir\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.469975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.469873 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:27.469975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.469911 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:27.469975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.469894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.469975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.469974 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:27.969956359 +0000 UTC m=+36.194579358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:27.470197 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.470013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.470197 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.470049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.470197 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.470111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178eccc4-c353-40e4-aec0-c2aa478cb2fe-config-volume\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.470389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.470368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.470897 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.470870 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.473944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.473924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.474039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.474022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a541b194-706b-4755-a04d-248fd6007a91-cert\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.474086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.474043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.480943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.480920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrbn\" (UniqueName: \"kubernetes.io/projected/178eccc4-c353-40e4-aec0-c2aa478cb2fe-kube-api-access-khrbn\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.481102 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.481076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whg68\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.481200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.481184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.481345 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.481321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhxf\" (UniqueName: \"kubernetes.io/projected/a541b194-706b-4755-a04d-248fd6007a91-kube-api-access-fmhxf\") pod \"ingress-canary-mdqkl\" (UID: \"a541b194-706b-4755-a04d-248fd6007a91\") " pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.505646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.505624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:27.590871 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.590844 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdqkl" Sep 15 21:16:27.974879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.974793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:27.975036 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:27.974882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:27.975036 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.974960 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:27.975036 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.975024 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:27.975147 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.975037 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:28.975016063 +0000 UTC m=+37.199639065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:27.975147 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.975045 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:27.975147 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:27.975098 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:28.975081857 +0000 UTC m=+37.199704856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:28.982768 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:28.982745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:28.982802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:28.982898 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:28.982908 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:28.982926 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:28.982938 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:30.982925971 +0000 UTC m=+39.207548966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:28.983524 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:28.982970 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:30.982954725 +0000 UTC m=+39.207577730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:29.048909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:29.048881 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdqkl"] Sep 15 21:16:29.056784 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:29.056757 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda541b194_706b_4755_a04d_248fd6007a91.slice/crio-184b0e9736708e959333e4f3285e011bc2f4f5b4b23d48c19b5d7291cc242a37 WatchSource:0}: Error finding container 184b0e9736708e959333e4f3285e011bc2f4f5b4b23d48c19b5d7291cc242a37: Status 404 returned error can't find the container with id 184b0e9736708e959333e4f3285e011bc2f4f5b4b23d48c19b5d7291cc242a37 Sep 15 21:16:29.459060 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:29.458887 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="c86ff5c496a6827628e49b07beb36cd5779f89043c886c314cd77f53f00b5d11" exitCode=0 Sep 15 21:16:29.459221 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:29.458963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"c86ff5c496a6827628e49b07beb36cd5779f89043c886c314cd77f53f00b5d11"} Sep 15 21:16:29.460327 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:29.460305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdqkl" event={"ID":"a541b194-706b-4755-a04d-248fd6007a91","Type":"ContainerStarted","Data":"184b0e9736708e959333e4f3285e011bc2f4f5b4b23d48c19b5d7291cc242a37"} Sep 15 21:16:30.465674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:30.465643 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e6c6be5-534c-4a3d-b29c-bd667defba9d" containerID="a029405aba6490c2a858e850ac5eaf624b27ccfb2ada1b82622a7a95f0454f43" exitCode=0 Sep 15 21:16:30.466091 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:30.465695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerDied","Data":"a029405aba6490c2a858e850ac5eaf624b27ccfb2ada1b82622a7a95f0454f43"} Sep 15 21:16:30.996508 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:30.996471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:30.996680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:30.996562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:30.996680 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:30.996586 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:30.996680 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:30.996609 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:30.996680 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:30.996670 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:34.996649855 +0000 UTC m=+43.221272858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:30.996680 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:30.996673 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:30.996881 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:30.996723 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:34.996706848 +0000 UTC m=+43.221329847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:31.363314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.363283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:31.428798 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.428760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:31.471322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.471288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkctb" event={"ID":"3e6c6be5-534c-4a3d-b29c-bd667defba9d","Type":"ContainerStarted","Data":"3df419d649b4820f693f843ccc724dab684bc75f4ca5abafff9bd7d85c55a028"} Sep 15 21:16:31.498117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.498067 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lkctb" podStartSLOduration=9.082469359 podStartE2EDuration="39.498051716s" totalImagesPullingTime="30.415582357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:31.497528518 +0000 UTC m=+39.722151523" watchObservedRunningTime="2026-09-15 21:16:31.498051716 +0000 UTC m=+39.722674719" Sep 15 21:16:31.618183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.618106 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:31.682054 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:31.682020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:32.903623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:32.903582 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:32.994591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:32.994564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:34.479744 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:34.479704 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdqkl" event={"ID":"a541b194-706b-4755-a04d-248fd6007a91","Type":"ContainerStarted","Data":"8193715bed7627a21942ee717b871c14f1abab4ea572a6f59f9f51a3e3c02826"} Sep 15 21:16:34.498064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:34.498015 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mdqkl" podStartSLOduration=3.140376532 podStartE2EDuration="7.498001693s" totalImagesPullingTime="4.357625161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.497449577 +0000 UTC m=+42.722072582" watchObservedRunningTime="2026-09-15 21:16:34.498001693 +0000 UTC m=+42.722624697" Sep 15 21:16:35.025368 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.025327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:35.025542 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.025418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:35.025542 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:35.025500 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:35.025610 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:35.025547 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:35.025610 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:35.025559 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:35.025610 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:35.025564 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:43.025549335 +0000 UTC m=+51.250172330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:35.025610 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:35.025599 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:43.025586618 +0000 UTC m=+51.250209616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:35.157705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.157662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:35.220054 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.220027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:35.440240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.440208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:35.541101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:35.541070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:36.359227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.359197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:36.960023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.959997 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6"] Sep 15 21:16:36.982198 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.982171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6"] Sep 15 21:16:36.982352 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.982268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:36.984875 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.984854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 21:16:36.984981 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.984934 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7j2zq\"" Sep 15 21:16:36.986105 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.986084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:36.986178 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:36.986087 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.039435 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.039411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.039549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.039437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5sp\" (UniqueName: \"kubernetes.io/projected/e3883551-7de9-4b29-aaab-98078d53cb23-kube-api-access-2p5sp\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.072307 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.072284 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9"] Sep 15 21:16:37.089972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.089952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.096752 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.096730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9"] Sep 15 21:16:37.096976 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.096951 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.096976 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.096969 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 15 21:16:37.097236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.096984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 15 21:16:37.097236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.096976 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.097412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.097397 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z2ch5\"" Sep 15 21:16:37.139919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.139893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.139919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.139919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5sp\" (UniqueName: \"kubernetes.io/projected/e3883551-7de9-4b29-aaab-98078d53cb23-kube-api-access-2p5sp\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.140090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.139953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d28eda98-9f3c-4c18-87ba-b0d65433ca10-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.140090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.140005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpnq\" (UniqueName: \"kubernetes.io/projected/d28eda98-9f3c-4c18-87ba-b0d65433ca10-kube-api-access-bzpnq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.140090 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.140030 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 21:16:37.140090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.140052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d28eda98-9f3c-4c18-87ba-b0d65433ca10-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.140269 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.140096 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls podName:e3883551-7de9-4b29-aaab-98078d53cb23 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.640081351 +0000 UTC m=+45.864704357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jjxl6" (UID: "e3883551-7de9-4b29-aaab-98078d53cb23") : secret "samples-operator-tls" not found Sep 15 21:16:37.156471 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.156427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5sp\" (UniqueName: \"kubernetes.io/projected/e3883551-7de9-4b29-aaab-98078d53cb23-kube-api-access-2p5sp\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.241026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.240967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d28eda98-9f3c-4c18-87ba-b0d65433ca10-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.241026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.241015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpnq\" (UniqueName: \"kubernetes.io/projected/d28eda98-9f3c-4c18-87ba-b0d65433ca10-kube-api-access-bzpnq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.241181 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.241061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d28eda98-9f3c-4c18-87ba-b0d65433ca10-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.241559 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.241539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d28eda98-9f3c-4c18-87ba-b0d65433ca10-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.243086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.243065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d28eda98-9f3c-4c18-87ba-b0d65433ca10-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.256379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.256353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpnq\" (UniqueName: \"kubernetes.io/projected/d28eda98-9f3c-4c18-87ba-b0d65433ca10-kube-api-access-bzpnq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-pxqx9\" (UID: \"d28eda98-9f3c-4c18-87ba-b0d65433ca10\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.281326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.281302 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx"] Sep 15 21:16:37.307860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.307841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx"] Sep 15 21:16:37.307943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.307916 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.310732 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.310706 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 15 21:16:37.310796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.310774 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-84cnj\"" Sep 15 21:16:37.310837 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.310788 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 15 21:16:37.319474 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.319438 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2"] Sep 15 21:16:37.341742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.341717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.341828 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.341783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.341965 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.341950 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2"] Sep 15 21:16:37.342050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.342038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.344908 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.344891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ps77j\"" Sep 15 21:16:37.344994 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.344943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:37.345166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.345141 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 15 21:16:37.345237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.345222 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 15 21:16:37.345294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.345263 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 15 21:16:37.379659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.379623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:37.398562 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.398542 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" Sep 15 21:16:37.443030 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.443151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e61f4ad-7e34-41e9-b647-cdb50515e398-serving-cert\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.443151 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.443137 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:37.443243 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84s6\" (UniqueName: \"kubernetes.io/projected/6e61f4ad-7e34-41e9-b647-cdb50515e398-kube-api-access-l84s6\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.443243 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.443196 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:16:37.943177349 +0000 UTC m=+46.167800351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:37.443349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.443349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e61f4ad-7e34-41e9-b647-cdb50515e398-config\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.443859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.443840 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.450612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.450592 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:37.543943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.543856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e61f4ad-7e34-41e9-b647-cdb50515e398-serving-cert\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.543943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.543916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84s6\" (UniqueName: \"kubernetes.io/projected/6e61f4ad-7e34-41e9-b647-cdb50515e398-kube-api-access-l84s6\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.544114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.544075 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e61f4ad-7e34-41e9-b647-cdb50515e398-config\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.544625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.544602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e61f4ad-7e34-41e9-b647-cdb50515e398-config\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.546054 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.546037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e61f4ad-7e34-41e9-b647-cdb50515e398-serving-cert\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.557554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.557535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84s6\" (UniqueName: \"kubernetes.io/projected/6e61f4ad-7e34-41e9-b647-cdb50515e398-kube-api-access-l84s6\") pod \"service-ca-operator-cf7c776b7-lssg2\" (UID: \"6e61f4ad-7e34-41e9-b647-cdb50515e398\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.645079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.645051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:37.645212 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.645187 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 21:16:37.645253 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.645237 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls podName:e3883551-7de9-4b29-aaab-98078d53cb23 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.645224214 +0000 UTC m=+46.869847208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jjxl6" (UID: "e3883551-7de9-4b29-aaab-98078d53cb23") : secret "samples-operator-tls" not found Sep 15 21:16:37.650044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.650027 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" Sep 15 21:16:37.687680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.687649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9"] Sep 15 21:16:37.690728 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:37.690699 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28eda98_9f3c_4c18_87ba_b0d65433ca10.slice/crio-6eac7d96fd6e8036a22b046699e755009b6f9510bb3e385c08aaffb09b52daa4 WatchSource:0}: Error finding container 6eac7d96fd6e8036a22b046699e755009b6f9510bb3e385c08aaffb09b52daa4: Status 404 returned error can't find the container with id 6eac7d96fd6e8036a22b046699e755009b6f9510bb3e385c08aaffb09b52daa4 Sep 15 21:16:37.946942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.946910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:37.947106 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.947090 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:37.947197 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:37.947182 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:16:38.947162245 +0000 UTC m=+47.171785244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:37.951565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:37.951542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2"] Sep 15 21:16:37.954484 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:37.954451 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e61f4ad_7e34_41e9_b647_cdb50515e398.slice/crio-cd1942a33ef1cdcfc42f55f24d1d5c45a5770eb87818fa41be7a2bccf2f0fb29 WatchSource:0}: Error finding container cd1942a33ef1cdcfc42f55f24d1d5c45a5770eb87818fa41be7a2bccf2f0fb29: Status 404 returned error can't find the container with id cd1942a33ef1cdcfc42f55f24d1d5c45a5770eb87818fa41be7a2bccf2f0fb29 Sep 15 21:16:38.491006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:38.490970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" event={"ID":"6e61f4ad-7e34-41e9-b647-cdb50515e398","Type":"ContainerStarted","Data":"cd1942a33ef1cdcfc42f55f24d1d5c45a5770eb87818fa41be7a2bccf2f0fb29"} Sep 15 21:16:38.492266 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:38.492241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" event={"ID":"d28eda98-9f3c-4c18-87ba-b0d65433ca10","Type":"ContainerStarted","Data":"6eac7d96fd6e8036a22b046699e755009b6f9510bb3e385c08aaffb09b52daa4"} Sep 15 21:16:38.652417 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:38.652352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:38.652553 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:38.652517 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 21:16:38.652608 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:38.652581 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls podName:e3883551-7de9-4b29-aaab-98078d53cb23 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:40.652562946 +0000 UTC m=+48.877185951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jjxl6" (UID: "e3883551-7de9-4b29-aaab-98078d53cb23") : secret "samples-operator-tls" not found Sep 15 21:16:38.954742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:38.954657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:38.954902 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:38.954825 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:38.954968 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:38.954907 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:16:40.954885807 +0000 UTC m=+49.179508802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:39.254059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:39.253980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:39.355690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:39.355660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:40.668591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:40.668559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:40.668915 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:40.668676 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 21:16:40.668915 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:40.668726 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls podName:e3883551-7de9-4b29-aaab-98078d53cb23 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.668712311 +0000 UTC m=+52.893335309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jjxl6" (UID: "e3883551-7de9-4b29-aaab-98078d53cb23") : secret "samples-operator-tls" not found Sep 15 21:16:40.971296 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:40.971221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:40.971427 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:40.971344 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:40.971427 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:40.971394 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.971380983 +0000 UTC m=+53.196003964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:41.502350 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:41.502307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" event={"ID":"6e61f4ad-7e34-41e9-b647-cdb50515e398","Type":"ContainerStarted","Data":"616d5122dfb3c6bc1990c4f6dd9df21e65b5f117ada49c3a36d5341c4880a006"} Sep 15 21:16:41.503959 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:41.503932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" event={"ID":"d28eda98-9f3c-4c18-87ba-b0d65433ca10","Type":"ContainerStarted","Data":"ffc0b4c2dd561ce88684755c49035a9fe0d40692d0d84e583a5a861cf5ca0122"} Sep 15 21:16:41.519405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:41.519360 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" podStartSLOduration=1.695433855 podStartE2EDuration="4.519347189s" totalImagesPullingTime="2.823913334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:41.518772384 +0000 UTC m=+49.743395399" watchObservedRunningTime="2026-09-15 21:16:41.519347189 +0000 UTC m=+49.743970194" Sep 15 21:16:41.537664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:41.537623 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" podStartSLOduration=1.456217295 podStartE2EDuration="4.537609092s" totalImagesPullingTime="3.081391797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:41.536886298 +0000 UTC m=+49.761509304" watchObservedRunningTime="2026-09-15 21:16:41.537609092 +0000 UTC m=+49.762232095" Sep 15 21:16:42.163336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:42.163305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:42.261722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:42.261694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:42.345360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:42.345331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:42.736812 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:42.736779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:42.841752 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:42.841725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:43.022041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.021976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:43.089832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.089804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:43.089976 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.089864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:43.089976 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:43.089953 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 21:16:43.090044 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:43.090021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls podName:178eccc4-c353-40e4-aec0-c2aa478cb2fe nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.090006226 +0000 UTC m=+67.314629208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls") pod "dns-default-zdkrl" (UID: "178eccc4-c353-40e4-aec0-c2aa478cb2fe") : secret "dns-default-metrics-tls" not found Sep 15 21:16:43.090044 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:43.089956 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 21:16:43.090044 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:43.090041 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67884f7d68-p9zgz: secret "image-registry-tls" not found Sep 15 21:16:43.090179 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:43.090068 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls podName:8de739ea-eddf-4cb6-bb1d-0fbf60a0572c nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.090061372 +0000 UTC m=+67.314684354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls") pod "image-registry-67884f7d68-p9zgz" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c") : secret "image-registry-tls" not found Sep 15 21:16:43.155715 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.155678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:43.174681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.174655 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6"] Sep 15 21:16:43.183208 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.183184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" Sep 15 21:16:43.189725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.189211 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nz6bg\"" Sep 15 21:16:43.189725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.189349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6"] Sep 15 21:16:43.189725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.189502 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 15 21:16:43.189725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.189717 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 15 21:16:43.249086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.249063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:43.291040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.290977 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsb8\" (UniqueName: \"kubernetes.io/projected/d53b978a-e206-4f11-bb71-1ddae2e7083e-kube-api-access-6hsb8\") pod \"migrator-8b4d697c5-bs6k6\" (UID: \"d53b978a-e206-4f11-bb71-1ddae2e7083e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" Sep 15 21:16:43.387218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.387189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:43.392345 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.392317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsb8\" (UniqueName: \"kubernetes.io/projected/d53b978a-e206-4f11-bb71-1ddae2e7083e-kube-api-access-6hsb8\") pod \"migrator-8b4d697c5-bs6k6\" (UID: \"d53b978a-e206-4f11-bb71-1ddae2e7083e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" Sep 15 21:16:43.418992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.418973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsb8\" (UniqueName: \"kubernetes.io/projected/d53b978a-e206-4f11-bb71-1ddae2e7083e-kube-api-access-6hsb8\") pod \"migrator-8b4d697c5-bs6k6\" (UID: \"d53b978a-e206-4f11-bb71-1ddae2e7083e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" Sep 15 21:16:43.492275 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.492254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" Sep 15 21:16:43.767473 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:43.767428 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6"] Sep 15 21:16:43.770771 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:43.770735 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53b978a_e206_4f11_bb71_1ddae2e7083e.slice/crio-79a90b50a5a3a15f0f9f958dbec87c47d56298006302bec29eac5cac205548a5 WatchSource:0}: Error finding container 79a90b50a5a3a15f0f9f958dbec87c47d56298006302bec29eac5cac205548a5: Status 404 returned error can't find the container with id 79a90b50a5a3a15f0f9f958dbec87c47d56298006302bec29eac5cac205548a5 Sep 15 21:16:44.115001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.114963 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gscvr"] Sep 15 21:16:44.130150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.130126 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.133336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.133156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 21:16:44.133677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.133651 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 21:16:44.133863 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.133841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 21:16:44.134037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.134003 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gscvr"] Sep 15 21:16:44.134107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.134091 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 21:16:44.134606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.134590 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6jkf\"" Sep 15 21:16:44.162152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.162126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:44.198480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.198440 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.198806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.198496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f0f7c0-c210-4802-b1a7-972d5ba25315-data-volume\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.198806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.198609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhv5x\" (UniqueName: \"kubernetes.io/projected/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-api-access-jhv5x\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.198806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.198715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95f0f7c0-c210-4802-b1a7-972d5ba25315-crio-socket\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.198806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.198742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.249528 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.249504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:44.299622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95f0f7c0-c210-4802-b1a7-972d5ba25315-crio-socket\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f0f7c0-c210-4802-b1a7-972d5ba25315-data-volume\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhv5x\" (UniqueName: \"kubernetes.io/projected/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-api-access-jhv5x\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.299789 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95f0f7c0-c210-4802-b1a7-972d5ba25315-crio-socket\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.299879 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.299801 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:44.299879 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.299830 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.300058 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.299884 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.799866552 +0000 UTC m=+53.024489538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.300058 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.299918 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:44.799904189 +0000 UTC m=+53.024527171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : configmap "kube-rbac-proxy" not found Sep 15 21:16:44.300169 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.300112 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f0f7c0-c210-4802-b1a7-972d5ba25315-data-volume\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.321641 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.321623 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhv5x\" (UniqueName: \"kubernetes.io/projected/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-api-access-jhv5x\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.513643 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.513568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" event={"ID":"d53b978a-e206-4f11-bb71-1ddae2e7083e","Type":"ContainerStarted","Data":"79a90b50a5a3a15f0f9f958dbec87c47d56298006302bec29eac5cac205548a5"} Sep 15 21:16:44.646283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.646249 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9fwff"] Sep 15 21:16:44.659297 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.659267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.662311 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.662292 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 15 21:16:44.662449 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.662299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 15 21:16:44.663980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.663796 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tw6dd\"" Sep 15 21:16:44.663980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.663804 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 15 21:16:44.663980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.663844 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 15 21:16:44.665117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.665094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9fwff"] Sep 15 21:16:44.703467 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.703427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:44.703573 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.703479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-cabundle\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.703573 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.703567 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 21:16:44.703658 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.703626 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls podName:e3883551-7de9-4b29-aaab-98078d53cb23 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:52.703604354 +0000 UTC m=+60.928227352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jjxl6" (UID: "e3883551-7de9-4b29-aaab-98078d53cb23") : secret "samples-operator-tls" not found Sep 15 21:16:44.703658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.703641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-key\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.703728 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.703697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw86m\" (UniqueName: \"kubernetes.io/projected/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-kube-api-access-pw86m\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.733776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.733752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:44.804107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-cabundle\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.804238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.804238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-key\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.804238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:44.804238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw86m\" (UniqueName: \"kubernetes.io/projected/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-kube-api-access-pw86m\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.804384 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.804253 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:44.804384 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.804303 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.804384 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.804338 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.804316354 +0000 UTC m=+54.028939362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : configmap "kube-rbac-proxy" not found Sep 15 21:16:44.804515 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:44.804386 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.804368512 +0000 UTC m=+54.028991509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:44.804846 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.804831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-cabundle\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.806382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.806365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-signing-key\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.823964 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.823943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw86m\" (UniqueName: \"kubernetes.io/projected/481ae0e9-a57f-47f4-9ab4-0286e151c9c6-kube-api-access-pw86m\") pod \"service-ca-568869fdc5-9fwff\" (UID: \"481ae0e9-a57f-47f4-9ab4-0286e151c9c6\") " pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:44.835989 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.835944 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:44.970246 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:44.970210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9fwff" Sep 15 21:16:45.006326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.006294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:45.006472 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.006418 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:45.006528 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.006514 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:16:53.006492227 +0000 UTC m=+61.231115235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:45.398341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.398269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9fwff"] Sep 15 21:16:45.402885 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:45.402861 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ae0e9_a57f_47f4_9ab4_0286e151c9c6.slice/crio-8c76598baf61b2b5b7d8cb75ba40f2fd249fcae475a1fbdee7fa889e19b526af WatchSource:0}: Error finding container 8c76598baf61b2b5b7d8cb75ba40f2fd249fcae475a1fbdee7fa889e19b526af: Status 404 returned error can't find the container with id 8c76598baf61b2b5b7d8cb75ba40f2fd249fcae475a1fbdee7fa889e19b526af Sep 15 21:16:45.518208 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.518170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" event={"ID":"d53b978a-e206-4f11-bb71-1ddae2e7083e","Type":"ContainerStarted","Data":"cff836b0c1a01565619a61f3e9cd2c5eeb47130d938478d235ace354a9756dc7"} Sep 15 21:16:45.518345 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.518233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" event={"ID":"d53b978a-e206-4f11-bb71-1ddae2e7083e","Type":"ContainerStarted","Data":"eb3a262f1991596d66e3643083f46614f4d0ab3fbacadda1cba68fae809220b1"} Sep 15 21:16:45.519723 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.519697 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9fwff" event={"ID":"481ae0e9-a57f-47f4-9ab4-0286e151c9c6","Type":"ContainerStarted","Data":"3780cedf4e2baecd5970fde65b090d57e4f48f5ab3656272f583209d59b66682"} Sep 15 21:16:45.519833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.519743 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9fwff" event={"ID":"481ae0e9-a57f-47f4-9ab4-0286e151c9c6","Type":"ContainerStarted","Data":"8c76598baf61b2b5b7d8cb75ba40f2fd249fcae475a1fbdee7fa889e19b526af"} Sep 15 21:16:45.557927 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.557883 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bs6k6" podStartSLOduration=1.186099635 podStartE2EDuration="2.55786943s" totalImagesPullingTime="1.371769795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:45.557603468 +0000 UTC m=+53.782226473" watchObservedRunningTime="2026-09-15 21:16:45.55786943 +0000 UTC m=+53.782492434" Sep 15 21:16:45.723574 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.723502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:45.813652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.813616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:45.813804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.813683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:45.813804 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.813746 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:45.813804 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.813794 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:45.813945 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.813816 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:47.813796776 +0000 UTC m=+56.038419771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : configmap "kube-rbac-proxy" not found Sep 15 21:16:45.813945 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:45.813870 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:47.813854707 +0000 UTC m=+56.038477704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:45.837265 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:45.837242 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:46.253803 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:46.253767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:46.495627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:46.495601 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss58r_947e6bd7-0aef-424b-8c05-ad8ac9aa28e7/dns-node-resolver/0.log" Sep 15 21:16:46.998000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:46.997960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:47.108479 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.108425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:47.295685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.295614 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzm7t_b4b0234d-2913-4587-b2de-6e9d61fa7293/node-ca/0.log" Sep 15 21:16:47.367765 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.367712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:47.542188 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.542161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:47.830780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.830747 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:47.830975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.830863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:47.830975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:47.830895 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:47.830975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:47.830925 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:47.830975 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:47.830964 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:51.830948247 +0000 UTC m=+60.055571250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:47.831186 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:47.830981 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:51.830973809 +0000 UTC m=+60.055596790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : configmap "kube-rbac-proxy" not found Sep 15 21:16:47.874124 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:47.874097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:48.236963 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.236886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:48.285697 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.285674 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mdqkl_a541b194-706b-4755-a04d-248fd6007a91/serve-healthcheck-canary/0.log" Sep 15 21:16:48.488018 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.487963 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bs6k6_d53b978a-e206-4f11-bb71-1ddae2e7083e/migrator/0.log" Sep 15 21:16:48.685656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.685632 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bs6k6_d53b978a-e206-4f11-bb71-1ddae2e7083e/graceful-termination/0.log" Sep 15 21:16:48.821824 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.821767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:48.887321 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:48.887296 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-pxqx9_d28eda98-9f3c-4c18-87ba-b0d65433ca10/kube-storage-version-migrator-operator/0.log" Sep 15 21:16:49.218587 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:49.218553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:49.428002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:49.427974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:49.908701 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:49.908668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:50.487051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:50.487022 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:50.565525 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:50.565495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:50.743972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:50.743900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:50.844323 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:50.844295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.219116 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.219058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.414342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.414297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.517310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.517213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.762847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.762812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.862957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.862927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:51.863139 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.862976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:51.863139 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:51.863069 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 21:16:51.863139 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:51.863113 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 21:16:51.863305 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:51.863149 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.863127826 +0000 UTC m=+68.087750821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : configmap "kube-rbac-proxy" not found Sep 15 21:16:51.863305 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:51.863168 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls podName:95f0f7c0-c210-4802-b1a7-972d5ba25315 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:59.863159969 +0000 UTC m=+68.087782950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gscvr" (UID: "95f0f7c0-c210-4802-b1a7-972d5ba25315") : secret "insights-runtime-extractor-tls" not found Sep 15 21:16:51.865101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:51.865072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:52.230909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.230837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:52.448530 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.448504 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkd82" Sep 15 21:16:52.491097 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.489736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-9fwff" podStartSLOduration=8.489714335 podStartE2EDuration="8.489714335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:45.590771328 +0000 UTC m=+53.815394331" watchObservedRunningTime="2026-09-15 21:16:52.489714335 +0000 UTC m=+60.714337341" Sep 15 21:16:52.770195 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.770099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:52.772437 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.772411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3883551-7de9-4b29-aaab-98078d53cb23-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jjxl6\" (UID: \"e3883551-7de9-4b29-aaab-98078d53cb23\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:52.823161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.823127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:52.894085 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.894062 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7j2zq\"" Sep 15 21:16:52.901406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:52.901384 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" Sep 15 21:16:53.072627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.072540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:16:53.072762 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:53.072659 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 21:16:53.072762 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:16:53.072714 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert podName:e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee nodeName:}" failed. No retries permitted until 2026-09-15 21:17:09.072699248 +0000 UTC m=+77.297322246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jzcnx" (UID: "e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee") : secret "networking-console-plugin-cert" not found Sep 15 21:16:53.168499 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.168465 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6"] Sep 15 21:16:53.215850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.215817 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:53.337362 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.337341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:53.438383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.438357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:53.544233 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:53.544209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" event={"ID":"e3883551-7de9-4b29-aaab-98078d53cb23","Type":"ContainerStarted","Data":"21866fc33cf50c3e2ae8203fd20d770add1a4cf588fc9f93ef649d68436a6ce0"} Sep 15 21:16:54.020827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:54.020775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:54.417040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:54.417002 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:54.612800 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:54.612766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:55.251893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.251822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:55.552821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.552747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" event={"ID":"e3883551-7de9-4b29-aaab-98078d53cb23","Type":"ContainerStarted","Data":"94e06392310270b9363bc59b1e56d0fdd217efea6a025fffbde4e59a4a17696e"} Sep 15 21:16:55.552821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.552808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" event={"ID":"e3883551-7de9-4b29-aaab-98078d53cb23","Type":"ContainerStarted","Data":"d132950cde0da45df6baa59d235d6b51681a1959c5ab20c3d9b8f12ebf7f9036"} Sep 15 21:16:55.570840 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.570790 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jjxl6" podStartSLOduration=17.836740708 podStartE2EDuration="19.570770512s" totalImagesPullingTime="1.734029804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:55.570158288 +0000 UTC m=+63.794781291" watchObservedRunningTime="2026-09-15 21:16:55.570770512 +0000 UTC m=+63.795393515" Sep 15 21:16:55.629084 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.629060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:55.846100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:55.846059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:56.284100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:56.284025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:56.820417 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:56.820382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:57.025917 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:57.025841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:57.224587 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:57.224548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:57.388808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:57.388780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:57.620675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:57.620639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:57.903771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:57.903739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:58.015565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.015541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:58.018435 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.018418 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 21:16:58.028661 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.028645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29d6225-c7eb-4ea1-9aa2-e09416b0ff26-metrics-certs\") pod \"network-metrics-daemon-wbzv9\" (UID: \"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26\") " pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:58.115996 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.115972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:58.116095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.116014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:58.118719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.118700 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 21:16:58.118852 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.118836 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 21:16:58.128920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.128904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c426a60-e8a8-4486-88f5-af838b4007c0-original-pull-secret\") pod \"global-pull-secret-syncer-nckfc\" (UID: \"3c426a60-e8a8-4486-88f5-af838b4007c0\") " pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:58.130150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.130134 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 21:16:58.139803 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.139783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhbb\" (UniqueName: \"kubernetes.io/projected/7db231b3-6660-4efc-806d-33ce05c8d31a-kube-api-access-6xhbb\") pod \"network-check-target-d6fj4\" (UID: \"7db231b3-6660-4efc-806d-33ce05c8d31a\") " pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:58.200526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.200446 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\"" Sep 15 21:16:58.205915 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.205895 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\"" Sep 15 21:16:58.209081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.209064 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:16:58.209156 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.209095 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nckfc" Sep 15 21:16:58.214286 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.214271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbzv9" Sep 15 21:16:58.489847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.489767 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nckfc"] Sep 15 21:16:58.492657 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:58.492621 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c426a60_e8a8_4486_88f5_af838b4007c0.slice/crio-8822c4ccaaae0d052410ab3ec37108444c1f1058f07e86fe24cd252c2e13807f WatchSource:0}: Error finding container 8822c4ccaaae0d052410ab3ec37108444c1f1058f07e86fe24cd252c2e13807f: Status 404 returned error can't find the container with id 8822c4ccaaae0d052410ab3ec37108444c1f1058f07e86fe24cd252c2e13807f Sep 15 21:16:58.492793 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.492698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d6fj4"] Sep 15 21:16:58.495719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.495616 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbzv9"] Sep 15 21:16:58.495854 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:58.495831 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db231b3_6660_4efc_806d_33ce05c8d31a.slice/crio-7982f248f48cab08737a34cd12ba6826262b5204c8688e7a683ff6ff3f66b51c WatchSource:0}: Error finding container 7982f248f48cab08737a34cd12ba6826262b5204c8688e7a683ff6ff3f66b51c: Status 404 returned error can't find the container with id 7982f248f48cab08737a34cd12ba6826262b5204c8688e7a683ff6ff3f66b51c Sep 15 21:16:58.501255 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:58.501232 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29d6225_c7eb_4ea1_9aa2_e09416b0ff26.slice/crio-2883885b6453f0e69209909381904c8431c9b4f137d48e3323b81201b196d416 WatchSource:0}: Error finding container 2883885b6453f0e69209909381904c8431c9b4f137d48e3323b81201b196d416: Status 404 returned error can't find the container with id 2883885b6453f0e69209909381904c8431c9b4f137d48e3323b81201b196d416 Sep 15 21:16:58.562982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.562951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbzv9" event={"ID":"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26","Type":"ContainerStarted","Data":"2883885b6453f0e69209909381904c8431c9b4f137d48e3323b81201b196d416"} Sep 15 21:16:58.564066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.564041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d6fj4" event={"ID":"7db231b3-6660-4efc-806d-33ce05c8d31a","Type":"ContainerStarted","Data":"7982f248f48cab08737a34cd12ba6826262b5204c8688e7a683ff6ff3f66b51c"} Sep 15 21:16:58.565094 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:58.565073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nckfc" event={"ID":"3c426a60-e8a8-4486-88f5-af838b4007c0","Type":"ContainerStarted","Data":"8822c4ccaaae0d052410ab3ec37108444c1f1058f07e86fe24cd252c2e13807f"} Sep 15 21:16:59.125128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.125097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:59.125493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.125142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:59.127890 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.127862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178eccc4-c353-40e4-aec0-c2aa478cb2fe-metrics-tls\") pod \"dns-default-zdkrl\" (UID: \"178eccc4-c353-40e4-aec0-c2aa478cb2fe\") " pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:59.128002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.127987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"image-registry-67884f7d68-p9zgz\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:59.338523 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.338485 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzsgv\"" Sep 15 21:16:59.346516 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.346487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:16:59.370257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.370234 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ztstg\"" Sep 15 21:16:59.378519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.378444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zdkrl" Sep 15 21:16:59.571141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.571038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbzv9" event={"ID":"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26","Type":"ContainerStarted","Data":"449c3bb01356a959cae59b3eaf198e4463530d061521a91a8099cc8694e4428b"} Sep 15 21:16:59.632425 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.632369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:16:59.635642 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:59.635612 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de739ea_eddf_4cb6_bb1d_0fbf60a0572c.slice/crio-2581e5047bb4d28b77c51f6743a5dc6bb0b4451350cfbbe52694861641898ff4 WatchSource:0}: Error finding container 2581e5047bb4d28b77c51f6743a5dc6bb0b4451350cfbbe52694861641898ff4: Status 404 returned error can't find the container with id 2581e5047bb4d28b77c51f6743a5dc6bb0b4451350cfbbe52694861641898ff4 Sep 15 21:16:59.695872 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.695848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zdkrl"] Sep 15 21:16:59.699156 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:16:59.699125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178eccc4_c353_40e4_aec0_c2aa478cb2fe.slice/crio-499d1f57b71bfc9309375160ea80ea9ca0e466b65fb2c13260b40794c6946461 WatchSource:0}: Error finding container 499d1f57b71bfc9309375160ea80ea9ca0e466b65fb2c13260b40794c6946461: Status 404 returned error can't find the container with id 499d1f57b71bfc9309375160ea80ea9ca0e466b65fb2c13260b40794c6946461 Sep 15 21:16:59.931052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.930966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:59.931052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.931025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:59.931569 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.931532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95f0f7c0-c210-4802-b1a7-972d5ba25315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:16:59.933549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:16:59.933522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95f0f7c0-c210-4802-b1a7-972d5ba25315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gscvr\" (UID: \"95f0f7c0-c210-4802-b1a7-972d5ba25315\") " pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:17:00.042672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.042600 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6jkf\"" Sep 15 21:17:00.050357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.050330 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gscvr" Sep 15 21:17:00.340196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.340165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gscvr"] Sep 15 21:17:00.343706 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:17:00.343676 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f0f7c0_c210_4802_b1a7_972d5ba25315.slice/crio-0bd91116d26fb6bb8b9ac436e2c86a2ab4ef5ddf296ccbfa71ff6cdc58136d72 WatchSource:0}: Error finding container 0bd91116d26fb6bb8b9ac436e2c86a2ab4ef5ddf296ccbfa71ff6cdc58136d72: Status 404 returned error can't find the container with id 0bd91116d26fb6bb8b9ac436e2c86a2ab4ef5ddf296ccbfa71ff6cdc58136d72 Sep 15 21:17:00.578001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.577966 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbzv9" event={"ID":"b29d6225-c7eb-4ea1-9aa2-e09416b0ff26","Type":"ContainerStarted","Data":"94e7db2408e93c6c31d936d00871acc0f53cb732f6fdfec7b43d6140c9550309"} Sep 15 21:17:00.579910 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.579880 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdkrl" event={"ID":"178eccc4-c353-40e4-aec0-c2aa478cb2fe","Type":"ContainerStarted","Data":"499d1f57b71bfc9309375160ea80ea9ca0e466b65fb2c13260b40794c6946461"} Sep 15 21:17:00.581857 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.581791 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gscvr" event={"ID":"95f0f7c0-c210-4802-b1a7-972d5ba25315","Type":"ContainerStarted","Data":"e69920ed3d1c69f50e5c6ce004f92438b737d2534bb109fad16be7a6048b806f"} Sep 15 21:17:00.581857 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.581845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gscvr" event={"ID":"95f0f7c0-c210-4802-b1a7-972d5ba25315","Type":"ContainerStarted","Data":"0bd91116d26fb6bb8b9ac436e2c86a2ab4ef5ddf296ccbfa71ff6cdc58136d72"} Sep 15 21:17:00.583695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.583667 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" event={"ID":"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c","Type":"ContainerStarted","Data":"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e"} Sep 15 21:17:00.583790 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.583712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" event={"ID":"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c","Type":"ContainerStarted","Data":"2581e5047bb4d28b77c51f6743a5dc6bb0b4451350cfbbe52694861641898ff4"} Sep 15 21:17:00.584001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.583949 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:17:00.597819 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.597735 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wbzv9" podStartSLOduration=67.683776717 podStartE2EDuration="1m8.597717772s" totalImagesPullingTime="913.941055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:00.596427901 +0000 UTC m=+68.821050906" watchObservedRunningTime="2026-09-15 21:17:00.597717772 +0000 UTC m=+68.822340779" Sep 15 21:17:00.619024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:00.618983 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" podStartSLOduration=68.618969984 podStartE2EDuration="1m8.618969984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:00.617830894 +0000 UTC m=+68.842453899" watchObservedRunningTime="2026-09-15 21:17:00.618969984 +0000 UTC m=+68.843592979" Sep 15 21:17:02.595158 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.595131 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d6fj4" event={"ID":"7db231b3-6660-4efc-806d-33ce05c8d31a","Type":"ContainerStarted","Data":"284c427683edd87e68d03f4145314be47d1222e3df2fbf32a31f31a138dd7a0b"} Sep 15 21:17:02.595483 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.595447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:17:02.597139 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.597110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdkrl" event={"ID":"178eccc4-c353-40e4-aec0-c2aa478cb2fe","Type":"ContainerStarted","Data":"1f4348c6979e74ccce53d5d8625ce0dc27b298fc9b7c6c71f42f80177417d284"} Sep 15 21:17:02.624253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.624137 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d6fj4" podStartSLOduration=66.82894189 podStartE2EDuration="1m10.624122293s" totalImagesPullingTime="3.795180403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:02.622782285 +0000 UTC m=+70.847405288" watchObservedRunningTime="2026-09-15 21:17:02.624122293 +0000 UTC m=+70.848745297" Sep 15 21:17:02.902337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.902299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:02.974656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:02.974615 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:03.603267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:03.603230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdkrl" event={"ID":"178eccc4-c353-40e4-aec0-c2aa478cb2fe","Type":"ContainerStarted","Data":"572acb94d3805b84c990f8b6fd435bfaf091f82a5f9c2b3ae5716fb7dcfafbd9"} Sep 15 21:17:03.603692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:03.603339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zdkrl" Sep 15 21:17:03.605427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:03.605399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gscvr" event={"ID":"95f0f7c0-c210-4802-b1a7-972d5ba25315","Type":"ContainerStarted","Data":"8382c256db2fc84133722d625bc02eef305077e09c52af59bb6538011375bb9b"} Sep 15 21:17:03.637302 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:03.637249 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zdkrl" podStartSLOduration=34.048134619 podStartE2EDuration="36.637234903s" totalImagesPullingTime="2.589100284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:03.635657971 +0000 UTC m=+71.860280977" watchObservedRunningTime="2026-09-15 21:17:03.637234903 +0000 UTC m=+71.861857906" Sep 15 21:17:05.617744 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:05.617705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nckfc" event={"ID":"3c426a60-e8a8-4486-88f5-af838b4007c0","Type":"ContainerStarted","Data":"1bdda6dbf002a72c63436cb9e4a210a6f79f3f06039c3a281a42fe8b77714c6c"} Sep 15 21:17:06.623819 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:06.623781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gscvr" event={"ID":"95f0f7c0-c210-4802-b1a7-972d5ba25315","Type":"ContainerStarted","Data":"8db9e42cd81644c3a3ab68a7c976f4bea48e7dd2af8fd78579b6aaecda1b8649"} Sep 15 21:17:06.661673 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:06.661626 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gscvr" podStartSLOduration=17.190239259 podStartE2EDuration="22.661612032s" totalImagesPullingTime="5.471372773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:06.661152302 +0000 UTC m=+74.885775318" watchObservedRunningTime="2026-09-15 21:17:06.661612032 +0000 UTC m=+74.886235036" Sep 15 21:17:06.661983 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:06.661961 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nckfc" podStartSLOduration=66.997732177 podStartE2EDuration="1m13.661956657s" totalImagesPullingTime="6.66422448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:05.638395891 +0000 UTC m=+73.863018895" watchObservedRunningTime="2026-09-15 21:17:06.661956657 +0000 UTC m=+74.886579660" Sep 15 21:17:07.858101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.858072 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j"] Sep 15 21:17:07.860041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.860025 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:07.868051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.868035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 15 21:17:07.883593 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.883577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 15 21:17:07.883682 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.883661 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 21:17:07.918359 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.918340 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 21:17:07.937726 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.937704 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 21:17:07.937842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.937825 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 15 21:17:07.941205 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.941188 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 15 21:17:07.947478 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:07.947434 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j"] Sep 15 21:17:08.008480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008446 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.008554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.008554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.008635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgz9g\" (UniqueName: \"kubernetes.io/projected/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-kube-api-access-bgz9g\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.008635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.008695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.008649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgz9g\" (UniqueName: \"kubernetes.io/projected/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-kube-api-access-bgz9g\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109577 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.109627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.109610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.110076 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.110049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.111768 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.111740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.111864 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.111828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.111920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.111874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.111920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.111891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-ca\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.137269 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.137241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgz9g\" (UniqueName: \"kubernetes.io/projected/6bd5cd58-32a1-4a35-8699-1ff00c7d137f-kube-api-access-bgz9g\") pod \"cluster-proxy-proxy-agent-7d779855b5-gkq7j\" (UID: \"6bd5cd58-32a1-4a35-8699-1ff00c7d137f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.178214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.178172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" Sep 15 21:17:08.274297 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.274266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:08.454010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.453935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:08.472622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.472601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j"] Sep 15 21:17:08.474979 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:17:08.474952 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bd5cd58_32a1_4a35_8699_1ff00c7d137f.slice/crio-6c6e9745a17ac86112d5a277be6221df4166d3452c85a98868be3b38ccccbeac WatchSource:0}: Error finding container 6c6e9745a17ac86112d5a277be6221df4166d3452c85a98868be3b38ccccbeac: Status 404 returned error can't find the container with id 6c6e9745a17ac86112d5a277be6221df4166d3452c85a98868be3b38ccccbeac Sep 15 21:17:08.601371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.601345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:08.631267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.631237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" event={"ID":"6bd5cd58-32a1-4a35-8699-1ff00c7d137f","Type":"ContainerStarted","Data":"6c6e9745a17ac86112d5a277be6221df4166d3452c85a98868be3b38ccccbeac"} Sep 15 21:17:08.709379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.709322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:08.804381 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:08.804352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:09.115533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.115496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:17:09.118253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.118210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jzcnx\" (UID: \"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:17:09.265597 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.265556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:09.419307 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.419231 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-84cnj\"" Sep 15 21:17:09.426602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.426576 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" Sep 15 21:17:09.601235 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.601181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:09.713042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.712935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:09.738695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:09.738664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx"] Sep 15 21:17:09.742431 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:17:09.742399 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode656b5c0_cb8d_47bd_ab1a_c3d10cc3c9ee.slice/crio-6410ad75927f74cd848c7bb0cea24eab928dea7d647ebbb520e117e896f13161 WatchSource:0}: Error finding container 6410ad75927f74cd848c7bb0cea24eab928dea7d647ebbb520e117e896f13161: Status 404 returned error can't find the container with id 6410ad75927f74cd848c7bb0cea24eab928dea7d647ebbb520e117e896f13161 Sep 15 21:17:10.641240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:10.641210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" event={"ID":"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee","Type":"ContainerStarted","Data":"6410ad75927f74cd848c7bb0cea24eab928dea7d647ebbb520e117e896f13161"} Sep 15 21:17:12.650674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.650638 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" event={"ID":"6bd5cd58-32a1-4a35-8699-1ff00c7d137f","Type":"ContainerStarted","Data":"9e6a9c31317c71f92518bb96963d8c454b8d0e26908bddacd5d40b64b0c8a074"} Sep 15 21:17:12.651099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.650688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" event={"ID":"6bd5cd58-32a1-4a35-8699-1ff00c7d137f","Type":"ContainerStarted","Data":"c72102d9c011953c8bb451bd6da0256852a42c72e5220d2557e6f720b27da0fd"} Sep 15 21:17:12.651099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.650701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" event={"ID":"6bd5cd58-32a1-4a35-8699-1ff00c7d137f","Type":"ContainerStarted","Data":"98f14edee792b4fc7228c1226be24403b31c2fb62f17d75fab19223a8f8d8557"} Sep 15 21:17:12.652142 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.652118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" event={"ID":"e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee","Type":"ContainerStarted","Data":"49b2320c6ed43703e14cfdd1b39ba525949efa7309566a0185b03b82e97cc023"} Sep 15 21:17:12.672413 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.672345 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d779855b5-gkq7j" podStartSLOduration=1.899027039 podStartE2EDuration="5.672332925s" totalImagesPullingTime="3.773305886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:12.670922991 +0000 UTC m=+80.895545994" watchObservedRunningTime="2026-09-15 21:17:12.672332925 +0000 UTC m=+80.896955928" Sep 15 21:17:12.690583 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.690538 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jzcnx" podStartSLOduration=33.198217146 podStartE2EDuration="35.690525136s" totalImagesPullingTime="2.49230799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:12.690189438 +0000 UTC m=+80.914812442" watchObservedRunningTime="2026-09-15 21:17:12.690525136 +0000 UTC m=+80.915148140" Sep 15 21:17:12.862498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.862444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:12.929213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:12.929145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:13.488579 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:13.488519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:13.586621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:13.586587 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:13.611588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:13.611564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zdkrl" Sep 15 21:17:15.451157 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:15.451043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:15.563770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:15.563735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:15.680261 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:15.680233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:15.787932 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:15.787866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:18.376346 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:18.376303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:18.447414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:18.447386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:18.713209 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:18.713136 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:18.791873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:18.791846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:19.284114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:19.284088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:19.350930 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:19.350900 2559 patch_prober.go:28] interesting pod/image-registry-67884f7d68-p9zgz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 15 21:17:19.351074 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:19.350958 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:17:20.562736 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:20.562681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:20.628396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:20.628369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:21.594810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:21.594776 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:17:21.852367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:21.852285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:21.972780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:21.972742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:22.330096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:22.330062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:22.756211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:22.756141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:22.877016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:22.876991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:23.866109 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:23.866077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.088448 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.088411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.135718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.135635 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kf7fm"] Sep 15 21:17:24.141378 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.141351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.144336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.144301 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 21:17:24.144713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.144696 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 21:17:24.144893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.144878 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hp94g\"" Sep 15 21:17:24.145690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.145670 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 21:17:24.145690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.145689 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 21:17:24.145866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.145847 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 21:17:24.146136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.145968 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 21:17:24.181839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.181803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.323820 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-wtmp\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.323988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbmc\" (UniqueName: \"kubernetes.io/projected/da79fc27-ce03-4f6e-bec1-9562778075f1-kube-api-access-wzbmc\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.323988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-metrics-client-ca\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.323988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.323988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-sys\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.323988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.323960 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-textfile\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.324172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.324002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.324172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.324075 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-tls\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.324172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.324117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-root\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.399258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.399175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.425347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-textfile\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.425546 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.425546 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-tls\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.425672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-root\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.425785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-textfile\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.425888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-root\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-wtmp\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbmc\" (UniqueName: \"kubernetes.io/projected/da79fc27-ce03-4f6e-bec1-9562778075f1-kube-api-access-wzbmc\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.425980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-metrics-client-ca\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-sys\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-sys\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-wtmp\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426670 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.426790 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.426726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da79fc27-ce03-4f6e-bec1-9562778075f1-metrics-client-ca\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.428474 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.428431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-tls\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.428552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.428470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da79fc27-ce03-4f6e-bec1-9562778075f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.437037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.437013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbmc\" (UniqueName: \"kubernetes.io/projected/da79fc27-ce03-4f6e-bec1-9562778075f1-kube-api-access-wzbmc\") pod \"node-exporter-kf7fm\" (UID: \"da79fc27-ce03-4f6e-bec1-9562778075f1\") " pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.454835 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.454815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kf7fm" Sep 15 21:17:24.467719 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:17:24.467698 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda79fc27_ce03_4f6e_bec1_9562778075f1.slice/crio-0e16f804f06bd2dd85d01ee3a1dbdafdea2c950ae305190982437aef3cbc68db WatchSource:0}: Error finding container 0e16f804f06bd2dd85d01ee3a1dbdafdea2c950ae305190982437aef3cbc68db: Status 404 returned error can't find the container with id 0e16f804f06bd2dd85d01ee3a1dbdafdea2c950ae305190982437aef3cbc68db Sep 15 21:17:24.493086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.493061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.592256 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.592229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.693581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.693510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:24.695080 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.695055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf7fm" event={"ID":"da79fc27-ce03-4f6e-bec1-9562778075f1","Type":"ContainerStarted","Data":"0e16f804f06bd2dd85d01ee3a1dbdafdea2c950ae305190982437aef3cbc68db"} Sep 15 21:17:24.937096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:24.937060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:25.186145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:25.186115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:25.346051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:25.346013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:25.561634 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:25.561560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:26.703723 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:26.703691 2559 generic.go:356] "Generic (PLEG): container finished" podID="da79fc27-ce03-4f6e-bec1-9562778075f1" containerID="1d301565631a41e99cbdc202e3a988ee5a38208605c4f8823b8db385d0467925" exitCode=0 Sep 15 21:17:26.704069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:26.703747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf7fm" event={"ID":"da79fc27-ce03-4f6e-bec1-9562778075f1","Type":"ContainerDied","Data":"1d301565631a41e99cbdc202e3a988ee5a38208605c4f8823b8db385d0467925"} Sep 15 21:17:27.182783 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.182750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:27.270900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.270875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:27.480004 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.479926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:27.563199 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.563171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:27.709255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.709222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf7fm" event={"ID":"da79fc27-ce03-4f6e-bec1-9562778075f1","Type":"ContainerStarted","Data":"90626d603d4b3360fe55fca6b6d5888fc2b4f371b572dd3bafb355fbe3296458"} Sep 15 21:17:27.709618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.709271 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf7fm" event={"ID":"da79fc27-ce03-4f6e-bec1-9562778075f1","Type":"ContainerStarted","Data":"16b71425b99d56c8ca3a93135bf9af0bc5a328dd179a970c05fc4c8e4b526f1b"} Sep 15 21:17:27.720374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.720349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:27.730735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:27.730664 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kf7fm" podStartSLOduration=2.435571763 podStartE2EDuration="3.730645678s" totalImagesPullingTime="1.232766549s" totalInitContainerRuntime="62.307366ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:27.729693544 +0000 UTC m=+95.954316539" watchObservedRunningTime="2026-09-15 21:17:27.730645678 +0000 UTC m=+95.955268681" Sep 15 21:17:28.893873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:28.893838 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:29.072584 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:29.072551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:30.454086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.454056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:30.492080 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.492044 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:17:30.560284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.560255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:30.622642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.622621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:30.824547 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.824482 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:30.892310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:30.892283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.237956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.237919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.313876 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.313851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.391314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.391290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.665944 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.665915 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.808329 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.808301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:31.907373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:31.907343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.174673 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.174638 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.371216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.371186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.471536 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.471422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.586366 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.586332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.874809 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.874780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.943716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:32.943693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:33.191769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:33.191707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:33.608440 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:33.608416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d6fj4" Sep 15 21:17:36.737397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:36.737357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:36.837652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:36.837624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:40.770997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:40.770962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:40.847667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:40.847637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:41.794670 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:41.794626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:41.895326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:41.895293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:42.156417 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:42.156381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:42.256878 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:42.256844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:43.991215 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:43.991180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:44.091001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:44.090969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:47.207937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:47.207895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:47.323610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:47.323579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:47.887791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:47.887755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:47.988092 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:47.988058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:51.799727 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:51.799694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:51.864197 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:51.864168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:53.997656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:53.997624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:54.062000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:54.061971 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:55.510621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.510576 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerName="registry" containerID="cri-o://8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e" gracePeriod=30 Sep 15 21:17:55.749125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.749102 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:17:55.804150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.804079 2559 generic.go:356] "Generic (PLEG): container finished" podID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerID="8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e" exitCode=0 Sep 15 21:17:55.804150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.804122 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" Sep 15 21:17:55.804150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.804130 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" event={"ID":"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c","Type":"ContainerDied","Data":"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e"} Sep 15 21:17:55.804365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.804175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67884f7d68-p9zgz" event={"ID":"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c","Type":"ContainerDied","Data":"2581e5047bb4d28b77c51f6743a5dc6bb0b4451350cfbbe52694861641898ff4"} Sep 15 21:17:55.804365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.804208 2559 scope.go:117] "RemoveContainer" containerID="8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e" Sep 15 21:17:55.814529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.814507 2559 scope.go:117] "RemoveContainer" containerID="8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e" Sep 15 21:17:55.814937 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:17:55.814919 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e\": container with ID starting with 8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e not found: ID does not exist" containerID="8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e" Sep 15 21:17:55.815024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.814944 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e"} err="failed to get container status \"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e\": rpc error: code = NotFound desc = could not find container \"8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e\": container with ID starting with 8431ec06a7e4a419f8d7f1ab79364b69831c421651300ef8c86fc142903b8a1e not found: ID does not exist" Sep 15 21:17:55.834297 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834277 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834307 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834369 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834388 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834445 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834501 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whg68\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68\") pod \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\" (UID: \"8de739ea-eddf-4cb6-bb1d-0fbf60a0572c\") " Sep 15 21:17:55.834995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.834861 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:17:55.835134 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.835097 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:17:55.837213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.837175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68" (OuterVolumeSpecName: "kube-api-access-whg68") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "kube-api-access-whg68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:55.837328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.837254 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:55.837328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.837294 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:55.837328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.837317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:55.837520 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.837441 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:55.843097 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.843077 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" (UID: "8de739ea-eddf-4cb6-bb1d-0fbf60a0572c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:17:55.935326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935306 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-trusted-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935323 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-bound-sa-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935333 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-installation-pull-secrets\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935343 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-ca-trust-extracted\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935351 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-certificates\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935360 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whg68\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-kube-api-access-whg68\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935368 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-image-registry-private-configuration\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.935430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:55.935376 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c-registry-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:17:56.125438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:56.125405 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:17:56.128896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:56.128876 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67884f7d68-p9zgz"] Sep 15 21:17:56.288480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:56.288441 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" path="/var/lib/kubelet/pods/8de739ea-eddf-4cb6-bb1d-0fbf60a0572c/volumes" Sep 15 21:17:56.588790 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:56.588761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:56.649914 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:56.649880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:57.492614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:57.492578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:57.595557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:17:57.595522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:02.874919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:02.874886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:02.944687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:02.944651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:06.842216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:06.842184 2559 generic.go:356] "Generic (PLEG): container finished" podID="6e61f4ad-7e34-41e9-b647-cdb50515e398" containerID="616d5122dfb3c6bc1990c4f6dd9df21e65b5f117ada49c3a36d5341c4880a006" exitCode=0 Sep 15 21:18:06.842617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:06.842259 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" event={"ID":"6e61f4ad-7e34-41e9-b647-cdb50515e398","Type":"ContainerDied","Data":"616d5122dfb3c6bc1990c4f6dd9df21e65b5f117ada49c3a36d5341c4880a006"} Sep 15 21:18:06.842617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:06.842555 2559 scope.go:117] "RemoveContainer" containerID="616d5122dfb3c6bc1990c4f6dd9df21e65b5f117ada49c3a36d5341c4880a006" Sep 15 21:18:07.847473 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:07.847419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lssg2" event={"ID":"6e61f4ad-7e34-41e9-b647-cdb50515e398","Type":"ContainerStarted","Data":"89549a106a0338ad9de51e53cea0b6ac77bcbeff8396ccf1b31b6950c93dd3ae"} Sep 15 21:18:10.326191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:10.326156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:10.426753 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:10.426727 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:10.497264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:10.497232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:11.866775 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:11.866739 2559 generic.go:356] "Generic (PLEG): container finished" podID="d28eda98-9f3c-4c18-87ba-b0d65433ca10" containerID="ffc0b4c2dd561ce88684755c49035a9fe0d40692d0d84e583a5a861cf5ca0122" exitCode=0 Sep 15 21:18:11.867230 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:11.866819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" event={"ID":"d28eda98-9f3c-4c18-87ba-b0d65433ca10","Type":"ContainerDied","Data":"ffc0b4c2dd561ce88684755c49035a9fe0d40692d0d84e583a5a861cf5ca0122"} Sep 15 21:18:11.867299 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:11.867234 2559 scope.go:117] "RemoveContainer" containerID="ffc0b4c2dd561ce88684755c49035a9fe0d40692d0d84e583a5a861cf5ca0122" Sep 15 21:18:12.872208 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:12.872176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-pxqx9" event={"ID":"d28eda98-9f3c-4c18-87ba-b0d65433ca10","Type":"ContainerStarted","Data":"9ab47226cdde91e28dfcc2ef3dfb45cdccaad58df2620ebe7a6726ddf49c98ee"} Sep 15 21:18:43.496104 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:43.496049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:43.716109 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:43.716076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.090766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.090691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.158272 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.158243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.224759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.224715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.277017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.276987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.826571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.826533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:18:52.932376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:18:52.932335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:02.774050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:02.774015 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:02.875396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:02.875361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:02.954192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:02.954162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:26.761123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:26.761089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:58.807862 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:58.807826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:58.867703 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:19:58.867675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:10.712346 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:10.712305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:11.188894 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:11.188859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:15.433942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:15.433865 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:15.502707 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:15.502678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:17.870200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:17.870167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:17.970907 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:17.970871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:26.037320 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:26.037282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:26.093913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:26.093882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:26.147422 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:26.147389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:20:52.199803 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:20:52.199772 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 21:21:06.288246 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:06.288208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:16.616591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:16.616557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:16.710316 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:16.710288 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:43.587268 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:43.587224 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:44.423257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:44.423221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:47.402244 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:47.402165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:47.472653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:47.472627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:48.656232 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:48.656198 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:48.801519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:48.801489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:48.991931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:48.991863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:49.097616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.097589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:49.959135 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.959106 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wx7zn"] Sep 15 21:21:49.959549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.959337 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerName="registry" Sep 15 21:21:49.959549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.959348 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerName="registry" Sep 15 21:21:49.959549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.959398 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de739ea-eddf-4cb6-bb1d-0fbf60a0572c" containerName="registry" Sep 15 21:21:49.962075 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.962058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:49.964945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.964923 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 21:21:49.965043 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.964966 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 21:21:49.965985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.965969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nng94\"" Sep 15 21:21:49.978772 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.978749 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wx7zn"] Sep 15 21:21:49.979744 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.979723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-kube-api-access-tnp2t\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:49.979827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:49.979783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.080568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.080538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.080729 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.080589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-kube-api-access-tnp2t\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.091250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.091213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.091361 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.091324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f-kube-api-access-tnp2t\") pod \"cert-manager-webhook-6486ffd997-wx7zn\" (UID: \"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.270973 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.270897 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:50.541789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.541717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wx7zn"] Sep 15 21:21:50.545379 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:21:50.545350 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b6998f_a144_4ea4_8b9a_c2a7b1f2fd0f.slice/crio-d01539052c35cbf416b0528e339e95b9e0adb9221fe0ca5ebcbd248ea4af9d42 WatchSource:0}: Error finding container d01539052c35cbf416b0528e339e95b9e0adb9221fe0ca5ebcbd248ea4af9d42: Status 404 returned error can't find the container with id d01539052c35cbf416b0528e339e95b9e0adb9221fe0ca5ebcbd248ea4af9d42 Sep 15 21:21:50.547256 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.547237 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:21:50.611521 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:50.611491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" event={"ID":"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f","Type":"ContainerStarted","Data":"d01539052c35cbf416b0528e339e95b9e0adb9221fe0ca5ebcbd248ea4af9d42"} Sep 15 21:21:51.333552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:51.333505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:51.829368 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:51.829330 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:54.174256 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:54.174227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:54.235545 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:54.235517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:54.628151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:54.628119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" event={"ID":"60b6998f-a144-4ea4-8b9a-c2a7b1f2fd0f","Type":"ContainerStarted","Data":"e2b9d20c00b8e044d0a6a6ceb2b6399e3dc5b02023705fc49e9647c06d75c60b"} Sep 15 21:21:54.628331 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:54.628192 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:21:54.680183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:54.680135 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" podStartSLOduration=2.472445719 podStartE2EDuration="5.680120154s" totalImagesPullingTime="3.207674435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:54.678892119 +0000 UTC m=+362.903515123" watchObservedRunningTime="2026-09-15 21:21:54.680120154 +0000 UTC m=+362.904743157" Sep 15 21:21:58.943644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:21:58.943612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:00.634154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:00.634124 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wx7zn" Sep 15 21:22:01.500917 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:01.500858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:01.600332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:01.600295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:07.992684 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:07.992648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:08.092570 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:08.092543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:09.763668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:09.763637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:12.935853 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:12.935818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:13.032398 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:13.032367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:21.719411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:21.719374 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:21.815227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:21.815184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:22.450928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:22.450897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:22.534621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:22.534586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:22.839231 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:22.839201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.155494 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.155401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.299643 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.299611 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.304966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.304936 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m"] Sep 15 21:22:23.308149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.308130 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.313302 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.313283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p85r2\"" Sep 15 21:22:23.313440 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.313421 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 21:22:23.314142 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.314125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 21:22:23.315104 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.315086 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 21:22:23.316023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.316003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 21:22:23.335223 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.335203 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m"] Sep 15 21:22:23.401929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.401903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.401929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.401938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.402123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.401971 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.402123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.402026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.402123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.402094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccgv\" (UniqueName: \"kubernetes.io/projected/10f8ab2b-7f47-4b12-8b13-648b4e20820a-kube-api-access-mccgv\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.491730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.491665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.502686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.502664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.502755 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.502706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.502755 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.502722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.502857 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.502840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccgv\" (UniqueName: \"kubernetes.io/projected/10f8ab2b-7f47-4b12-8b13-648b4e20820a-kube-api-access-mccgv\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.502910 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.502895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.503238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.503214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.504992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.504963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f8ab2b-7f47-4b12-8b13-648b4e20820a-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.505142 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.505120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.505277 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.505263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f8ab2b-7f47-4b12-8b13-648b4e20820a-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.510986 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.510967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccgv\" (UniqueName: \"kubernetes.io/projected/10f8ab2b-7f47-4b12-8b13-648b4e20820a-kube-api-access-mccgv\") pod \"opendatahub-operator-controller-manager-596bc79d9b-p6g7m\" (UID: \"10f8ab2b-7f47-4b12-8b13-648b4e20820a\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.593210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.593189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.618143 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.618124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:23.694792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.694767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.797138 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.797071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.894742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.894678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m"] Sep 15 21:22:23.895262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:23.895193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:23.897801 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:22:23.897776 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f8ab2b_7f47_4b12_8b13_648b4e20820a.slice/crio-143f0053b6b94621e6cc40b1a89b4af8fe55e95bb31767b3e90a0ec787f96094 WatchSource:0}: Error finding container 143f0053b6b94621e6cc40b1a89b4af8fe55e95bb31767b3e90a0ec787f96094: Status 404 returned error can't find the container with id 143f0053b6b94621e6cc40b1a89b4af8fe55e95bb31767b3e90a0ec787f96094 Sep 15 21:22:24.002571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:24.002548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:24.730333 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:24.730257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" event={"ID":"10f8ab2b-7f47-4b12-8b13-648b4e20820a","Type":"ContainerStarted","Data":"143f0053b6b94621e6cc40b1a89b4af8fe55e95bb31767b3e90a0ec787f96094"} Sep 15 21:22:27.743908 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:27.743874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" event={"ID":"10f8ab2b-7f47-4b12-8b13-648b4e20820a","Type":"ContainerStarted","Data":"c1ebb380d56115c6daa336ddbb0739aaf998e2fedee07d1cf36ea4bd0e3726e4"} Sep 15 21:22:28.748766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:28.748730 2559 generic.go:356] "Generic (PLEG): container finished" podID="10f8ab2b-7f47-4b12-8b13-648b4e20820a" containerID="c1ebb380d56115c6daa336ddbb0739aaf998e2fedee07d1cf36ea4bd0e3726e4" exitCode=0 Sep 15 21:22:28.749156 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:28.748814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" event={"ID":"10f8ab2b-7f47-4b12-8b13-648b4e20820a","Type":"ContainerDied","Data":"c1ebb380d56115c6daa336ddbb0739aaf998e2fedee07d1cf36ea4bd0e3726e4"} Sep 15 21:22:29.753970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:29.753923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" event={"ID":"10f8ab2b-7f47-4b12-8b13-648b4e20820a","Type":"ContainerStarted","Data":"49a7c94899de9f4b9ce255a40fc4ddb9e9e6565f769af504d7da48d2425201dd"} Sep 15 21:22:29.754348 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:29.754057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:29.778796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:29.778747 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" podStartSLOduration=2.424017952 podStartE2EDuration="6.778732423s" totalImagesPullingTime="3.322095656s" totalInitContainerRuntime="1.032618815s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:29.777217368 +0000 UTC m=+398.001840384" watchObservedRunningTime="2026-09-15 21:22:29.778732423 +0000 UTC m=+398.003355426" Sep 15 21:22:33.960988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:33.960949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:34.056691 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:34.056645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:39.065611 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:39.065577 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:39.137302 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:39.137275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:40.759319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:40.759294 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-p6g7m" Sep 15 21:22:42.442498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.442449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:42.504987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.503130 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl"] Sep 15 21:22:42.518644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.518615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.523441 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.523419 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 15 21:22:42.523602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.523435 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 21:22:42.523761 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.523736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 21:22:42.523761 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.523736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 15 21:22:42.524041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.524024 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 15 21:22:42.524322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.524304 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lbk2h\"" Sep 15 21:22:42.524672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.524653 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl"] Sep 15 21:22:42.601731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.601661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:42.639544 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.639513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z26nc\" (UniqueName: \"kubernetes.io/projected/8a3a63fe-9659-434e-8863-ce99627131c2-kube-api-access-z26nc\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.639645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.639558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.639645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.639589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-metrics-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.639716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.639647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a3a63fe-9659-434e-8863-ce99627131c2-manager-config\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.740957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.740887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z26nc\" (UniqueName: \"kubernetes.io/projected/8a3a63fe-9659-434e-8863-ce99627131c2-kube-api-access-z26nc\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.740957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.740924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.740957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.740954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-metrics-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.741126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.740978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a3a63fe-9659-434e-8863-ce99627131c2-manager-config\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.741724 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.741704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a3a63fe-9659-434e-8863-ce99627131c2-manager-config\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.743393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.743371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.743479 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.743446 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a3a63fe-9659-434e-8863-ce99627131c2-metrics-cert\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.798771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.798747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z26nc\" (UniqueName: \"kubernetes.io/projected/8a3a63fe-9659-434e-8863-ce99627131c2-kube-api-access-z26nc\") pod \"lws-controller-manager-dccb565dd-dmsdl\" (UID: \"8a3a63fe-9659-434e-8863-ce99627131c2\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.828570 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.828545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:42.949393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:42.949357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:43.127207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:43.127181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl"] Sep 15 21:22:43.129640 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:22:43.129613 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3a63fe_9659_434e_8863_ce99627131c2.slice/crio-e28117d7c7687e00404da62e4d43282ea9403e09283a2f28d67f96eddaf27a61 WatchSource:0}: Error finding container e28117d7c7687e00404da62e4d43282ea9403e09283a2f28d67f96eddaf27a61: Status 404 returned error can't find the container with id e28117d7c7687e00404da62e4d43282ea9403e09283a2f28d67f96eddaf27a61 Sep 15 21:22:43.142890 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:43.142863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:43.802475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:43.802424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" event={"ID":"8a3a63fe-9659-434e-8863-ce99627131c2","Type":"ContainerStarted","Data":"e28117d7c7687e00404da62e4d43282ea9403e09283a2f28d67f96eddaf27a61"} Sep 15 21:22:44.753655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:44.753606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:44.843230 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:44.843191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:45.784507 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:45.784449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:46.814891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:46.814857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" event={"ID":"8a3a63fe-9659-434e-8863-ce99627131c2","Type":"ContainerStarted","Data":"c7193c7c0f77a2b5062b10e1f7b30a22e11e8dc6645d416bec17a3425044239a"} Sep 15 21:22:46.815344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:46.814972 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:46.832883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:46.832836 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" podStartSLOduration=2.115774382 podStartE2EDuration="4.832822334s" totalImagesPullingTime="2.717047952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:46.831165159 +0000 UTC m=+415.055788163" watchObservedRunningTime="2026-09-15 21:22:46.832822334 +0000 UTC m=+415.057445337" Sep 15 21:22:49.692114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:49.692074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:50.061318 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.061233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:50.405130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.405098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:50.636710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.636680 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx"] Sep 15 21:22:50.639920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.639901 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.643085 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.643066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 21:22:50.643171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.643101 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mzcqj\"" Sep 15 21:22:50.643171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.643073 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 21:22:50.643294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.643201 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 21:22:50.658636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.658584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx"] Sep 15 21:22:50.670015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.669991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:50.799760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c95cc0-5431-4388-81b1-b30d357e0873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c95cc0-5431-4388-81b1-b30d357e0873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmlp\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-kube-api-access-lzmlp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.800081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.799942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901157 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c95cc0-5431-4388-81b1-b30d357e0873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c95cc0-5431-4388-81b1-b30d357e0873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmlp\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-kube-api-access-lzmlp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901677 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.901851 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.901833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.902095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.902076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c95cc0-5431-4388-81b1-b30d357e0873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.903563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.903538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c95cc0-5431-4388-81b1-b30d357e0873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.904067 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.904048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c95cc0-5431-4388-81b1-b30d357e0873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.909654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.909602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.909981 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.909952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmlp\" (UniqueName: \"kubernetes.io/projected/29c95cc0-5431-4388-81b1-b30d357e0873-kube-api-access-lzmlp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx\" (UID: \"29c95cc0-5431-4388-81b1-b30d357e0873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:50.950563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:50.950539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:51.035325 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:51.035296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:51.089261 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:51.089236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:51.231300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:51.231243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx"] Sep 15 21:22:51.234749 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:22:51.234712 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c95cc0_5431_4388_81b1_b30d357e0873.slice/crio-882d02a4bd153bd2b6aae9aa2e00b924a07468e1237c152cbe8b89876c612969 WatchSource:0}: Error finding container 882d02a4bd153bd2b6aae9aa2e00b924a07468e1237c152cbe8b89876c612969: Status 404 returned error can't find the container with id 882d02a4bd153bd2b6aae9aa2e00b924a07468e1237c152cbe8b89876c612969 Sep 15 21:22:51.835059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:51.835025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" event={"ID":"29c95cc0-5431-4388-81b1-b30d357e0873","Type":"ContainerStarted","Data":"882d02a4bd153bd2b6aae9aa2e00b924a07468e1237c152cbe8b89876c612969"} Sep 15 21:22:53.430981 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.430946 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-srhr2"] Sep 15 21:22:53.437787 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.437758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.441581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.441559 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 15 21:22:53.441836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.441815 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-grtt8\"" Sep 15 21:22:53.448093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.448073 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-srhr2"] Sep 15 21:22:53.506038 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.505967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:53.522602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.522573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.522731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.522668 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmtg\" (UniqueName: \"kubernetes.io/projected/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-kube-api-access-wdmtg\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.609805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.609770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:53.623976 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.623944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.624104 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.624043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmtg\" (UniqueName: \"kubernetes.io/projected/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-kube-api-access-wdmtg\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.624167 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:22:53.624115 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 21:22:53.624218 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:22:53.624188 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert podName:e258e6cf-526f-46fa-9a5f-c7ef5ae094ac nodeName:}" failed. No retries permitted until 2026-09-15 21:22:54.124168904 +0000 UTC m=+422.348791909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert") pod "odh-model-controller-7998c67dc4-srhr2" (UID: "e258e6cf-526f-46fa-9a5f-c7ef5ae094ac") : secret "odh-model-controller-webhook-cert" not found Sep 15 21:22:53.639744 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.639720 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmtg\" (UniqueName: \"kubernetes.io/projected/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-kube-api-access-wdmtg\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:53.861629 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.861599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:53.879994 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.879958 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:22:53.880061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.880042 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:22:53.880096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:53.880074 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:22:54.020105 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.020073 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jnx9r"] Sep 15 21:22:54.023237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.023220 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.026020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.025998 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 15 21:22:54.034054 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.034029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jnx9r"] Sep 15 21:22:54.041192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.041165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:54.129602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.129538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:54.129602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.129581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f049b212-f487-4f01-858d-3ff00bb80425-tls-certs\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.129760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.129623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544w2\" (UniqueName: \"kubernetes.io/projected/f049b212-f487-4f01-858d-3ff00bb80425-kube-api-access-544w2\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.131964 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.131939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e258e6cf-526f-46fa-9a5f-c7ef5ae094ac-cert\") pod \"odh-model-controller-7998c67dc4-srhr2\" (UID: \"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac\") " pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:54.230371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.230334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544w2\" (UniqueName: \"kubernetes.io/projected/f049b212-f487-4f01-858d-3ff00bb80425-kube-api-access-544w2\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.230589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.230424 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f049b212-f487-4f01-858d-3ff00bb80425-tls-certs\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.232789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.232768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f049b212-f487-4f01-858d-3ff00bb80425-tls-certs\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.239255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.239232 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-544w2\" (UniqueName: \"kubernetes.io/projected/f049b212-f487-4f01-858d-3ff00bb80425-kube-api-access-544w2\") pod \"model-serving-api-54d89bd5fc-jnx9r\" (UID: \"f049b212-f487-4f01-858d-3ff00bb80425\") " pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.251841 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.251813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:54.334335 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.334306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:54.349356 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.349330 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:54.604120 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.604066 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jnx9r"] Sep 15 21:22:54.606586 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:22:54.606559 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf049b212_f487_4f01_858d_3ff00bb80425.slice/crio-bf182dc8a6c1d885981d82f9ef7f0d0e5f1825c9e383b03dd3860fa2c8c75255 WatchSource:0}: Error finding container bf182dc8a6c1d885981d82f9ef7f0d0e5f1825c9e383b03dd3860fa2c8c75255: Status 404 returned error can't find the container with id bf182dc8a6c1d885981d82f9ef7f0d0e5f1825c9e383b03dd3860fa2c8c75255 Sep 15 21:22:54.632625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.632532 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-srhr2"] Sep 15 21:22:54.634897 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:22:54.634874 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode258e6cf_526f_46fa_9a5f_c7ef5ae094ac.slice/crio-29a6d3d6fa21fe3cbce3950271dbb953346cf5d12ee9badd73554067739e5c43 WatchSource:0}: Error finding container 29a6d3d6fa21fe3cbce3950271dbb953346cf5d12ee9badd73554067739e5c43: Status 404 returned error can't find the container with id 29a6d3d6fa21fe3cbce3950271dbb953346cf5d12ee9badd73554067739e5c43 Sep 15 21:22:54.848972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.848941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" event={"ID":"29c95cc0-5431-4388-81b1-b30d357e0873","Type":"ContainerStarted","Data":"e9eabf906cf3feec723eace7e167fa1a3db081f49cdbf0d5d90c267267cd9643"} Sep 15 21:22:54.850241 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.850215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" event={"ID":"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac","Type":"ContainerStarted","Data":"29a6d3d6fa21fe3cbce3950271dbb953346cf5d12ee9badd73554067739e5c43"} Sep 15 21:22:54.851228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.851203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" event={"ID":"f049b212-f487-4f01-858d-3ff00bb80425","Type":"ContainerStarted","Data":"bf182dc8a6c1d885981d82f9ef7f0d0e5f1825c9e383b03dd3860fa2c8c75255"} Sep 15 21:22:54.868942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.868901 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" podStartSLOduration=2.225884819 podStartE2EDuration="4.868886768s" totalImagesPullingTime="2.643001949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:54.867189542 +0000 UTC m=+423.091812547" watchObservedRunningTime="2026-09-15 21:22:54.868886768 +0000 UTC m=+423.093509772" Sep 15 21:22:54.951080 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.951024 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:54.955422 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:54.955402 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:55.858436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:55.858405 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:55.859505 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:55.859482 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx" Sep 15 21:22:57.820595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:57.820560 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-dmsdl" Sep 15 21:22:58.874432 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.874393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" event={"ID":"e258e6cf-526f-46fa-9a5f-c7ef5ae094ac","Type":"ContainerStarted","Data":"359600f99b0806ed9e5240c65599602bb84b796d24d37175762331b0035f684b"} Sep 15 21:22:58.874828 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.874508 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:22:58.876274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.876247 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" event={"ID":"f049b212-f487-4f01-858d-3ff00bb80425","Type":"ContainerStarted","Data":"882d2ac14b76b539109fb7aab38027dced5687da9b9fe8818cda539b1f452f25"} Sep 15 21:22:58.876398 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.876375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:22:58.895777 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.895736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" podStartSLOduration=2.108642106 podStartE2EDuration="5.895722279s" totalImagesPullingTime="3.787080173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:58.894242515 +0000 UTC m=+427.118865522" watchObservedRunningTime="2026-09-15 21:22:58.895722279 +0000 UTC m=+427.120345284" Sep 15 21:22:58.919250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:22:58.919205 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" podStartSLOduration=2.152646558 podStartE2EDuration="5.91919269s" totalImagesPullingTime="3.766546132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:58.917643562 +0000 UTC m=+427.142266565" watchObservedRunningTime="2026-09-15 21:22:58.91919269 +0000 UTC m=+427.143815694" Sep 15 21:23:03.953026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:03.952979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:04.071489 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.071441 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf"] Sep 15 21:23:04.074444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.074427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.079686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.079661 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 15 21:23:04.080243 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.080223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vmtfw\"" Sep 15 21:23:04.094428 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.094399 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf"] Sep 15 21:23:04.094734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.094714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:04.209475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.209374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94f4cffc-2549-4841-9c76-5925407909f1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.209475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.209413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znwr\" (UniqueName: \"kubernetes.io/projected/94f4cffc-2549-4841-9c76-5925407909f1-kube-api-access-6znwr\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.229696 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.229671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:04.310577 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.310553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znwr\" (UniqueName: \"kubernetes.io/projected/94f4cffc-2549-4841-9c76-5925407909f1-kube-api-access-6znwr\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.310691 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.310623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94f4cffc-2549-4841-9c76-5925407909f1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.312987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.312961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94f4cffc-2549-4841-9c76-5925407909f1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.336757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.336728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znwr\" (UniqueName: \"kubernetes.io/projected/94f4cffc-2549-4841-9c76-5925407909f1-kube-api-access-6znwr\") pod \"llmisvc-controller-manager-777c6bd7d5-547vf\" (UID: \"94f4cffc-2549-4841-9c76-5925407909f1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.385404 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.385369 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:04.545058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.544980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:04.663172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.663145 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf"] Sep 15 21:23:04.666804 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:23:04.666775 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94f4cffc_2549_4841_9c76_5925407909f1.slice/crio-128bdd7b47cca99c4c60d261ce60241e14320a251d02aea2407c0471454610cb WatchSource:0}: Error finding container 128bdd7b47cca99c4c60d261ce60241e14320a251d02aea2407c0471454610cb: Status 404 returned error can't find the container with id 128bdd7b47cca99c4c60d261ce60241e14320a251d02aea2407c0471454610cb Sep 15 21:23:04.901335 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:04.901304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" event={"ID":"94f4cffc-2549-4841-9c76-5925407909f1","Type":"ContainerStarted","Data":"128bdd7b47cca99c4c60d261ce60241e14320a251d02aea2407c0471454610cb"} Sep 15 21:23:05.363932 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:05.363899 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:05.505013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:05.504979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:07.915351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:07.915325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" event={"ID":"94f4cffc-2549-4841-9c76-5925407909f1","Type":"ContainerStarted","Data":"d2a481ca8abbfc0584f8d16a7a559f24a143c6a04deefa22ad9834d1379de0b9"} Sep 15 21:23:07.915654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:07.915398 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:07.938392 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:07.938351 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" podStartSLOduration=0.878320444 podStartE2EDuration="3.938333041s" totalImagesPullingTime="3.060012597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:07.937096734 +0000 UTC m=+436.161719739" watchObservedRunningTime="2026-09-15 21:23:07.938333041 +0000 UTC m=+436.162956046" Sep 15 21:23:09.882227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:09.882197 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-srhr2" Sep 15 21:23:09.884189 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:09.884168 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jnx9r" Sep 15 21:23:38.921305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:38.921269 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-547vf" Sep 15 21:23:39.711928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:39.711890 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:41.657208 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:41.657170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:41.732129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:41.732095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:54.573706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:54.573665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:23:54.676626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:23:54.676590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:10.841719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:10.841682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:12.889444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.889408 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79"] Sep 15 21:24:12.892849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.892832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:12.896812 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.896787 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 21:24:12.896948 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.896869 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 21:24:12.897112 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.897098 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bp9ll\"" Sep 15 21:24:12.897286 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.897274 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 15 21:24:12.908397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:12.908371 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79"] Sep 15 21:24:13.036705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.036664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxhs\" (UniqueName: \"kubernetes.io/projected/f6e43b9c-f6ab-4b0d-a466-b5bdef85714f-kube-api-access-2qxhs\") pod \"dns-operator-controller-manager-7b84cd7c9c-8bg79\" (UID: \"f6e43b9c-f6ab-4b0d-a466-b5bdef85714f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:13.137198 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.137166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxhs\" (UniqueName: \"kubernetes.io/projected/f6e43b9c-f6ab-4b0d-a466-b5bdef85714f-kube-api-access-2qxhs\") pod \"dns-operator-controller-manager-7b84cd7c9c-8bg79\" (UID: \"f6e43b9c-f6ab-4b0d-a466-b5bdef85714f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:13.150403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.150355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxhs\" (UniqueName: \"kubernetes.io/projected/f6e43b9c-f6ab-4b0d-a466-b5bdef85714f-kube-api-access-2qxhs\") pod \"dns-operator-controller-manager-7b84cd7c9c-8bg79\" (UID: \"f6e43b9c-f6ab-4b0d-a466-b5bdef85714f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:13.202975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.202949 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:13.485737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.485673 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79"] Sep 15 21:24:13.488529 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:13.488498 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e43b9c_f6ab_4b0d_a466_b5bdef85714f.slice/crio-fcaf07b80dec0a2411ca4a1fe9457e15f847069e0c7df7532a7fb4a742a0cde1 WatchSource:0}: Error finding container fcaf07b80dec0a2411ca4a1fe9457e15f847069e0c7df7532a7fb4a742a0cde1: Status 404 returned error can't find the container with id fcaf07b80dec0a2411ca4a1fe9457e15f847069e0c7df7532a7fb4a742a0cde1 Sep 15 21:24:13.689895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:13.689861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:14.184030 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:14.183994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" event={"ID":"f6e43b9c-f6ab-4b0d-a466-b5bdef85714f","Type":"ContainerStarted","Data":"fcaf07b80dec0a2411ca4a1fe9457e15f847069e0c7df7532a7fb4a742a0cde1"} Sep 15 21:24:15.510887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:15.510852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:15.649546 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:15.649519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:16.195523 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.195422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" event={"ID":"f6e43b9c-f6ab-4b0d-a466-b5bdef85714f","Type":"ContainerStarted","Data":"4ff1dd82fc6151d206418cc6b3bd63ec5c58815a25e258587e7f603af230412b"} Sep 15 21:24:16.195676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.195538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:16.218520 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.218446 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" podStartSLOduration=1.7835618389999999 podStartE2EDuration="4.21843282s" totalImagesPullingTime="2.434870981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:16.218014451 +0000 UTC m=+504.442637456" watchObservedRunningTime="2026-09-15 21:24:16.21843282 +0000 UTC m=+504.443055824" Sep 15 21:24:16.248970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.248942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:16.619136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.619105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:16.938884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:16.938803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:17.722212 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:17.722179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:17.822710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:17.822684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:19.842365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.842334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:19.921794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.921766 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt"] Sep 15 21:24:19.924936 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.924920 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:19.928028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.928012 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-45z78\"" Sep 15 21:24:19.950403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.950378 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt"] Sep 15 21:24:19.987538 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.987514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8m6b\" (UniqueName: \"kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:19.987628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:19.987582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.088563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.088528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8m6b\" (UniqueName: \"kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.088722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.088689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.089011 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.088986 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.102841 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.102794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8m6b\" (UniqueName: \"kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.234348 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.234316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:20.252614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.252586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:20.524701 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:20.524673 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt"] Sep 15 21:24:20.525752 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:20.525717 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441f2572_ce20_415a_8e06_601d6787e3d0.slice/crio-918d562e79b2f0788198c92a816c2ced67c280757a972b2ab49aee97c41992f7 WatchSource:0}: Error finding container 918d562e79b2f0788198c92a816c2ced67c280757a972b2ab49aee97c41992f7: Status 404 returned error can't find the container with id 918d562e79b2f0788198c92a816c2ced67c280757a972b2ab49aee97c41992f7 Sep 15 21:24:21.216736 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:21.216698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" event={"ID":"441f2572-ce20-415a-8e06-601d6787e3d0","Type":"ContainerStarted","Data":"918d562e79b2f0788198c92a816c2ced67c280757a972b2ab49aee97c41992f7"} Sep 15 21:24:21.701885 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:21.701819 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:23.447009 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:23.446969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:23.661309 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:23.661264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:23.758124 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:23.758018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:25.237529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:25.237487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" event={"ID":"441f2572-ce20-415a-8e06-601d6787e3d0","Type":"ContainerStarted","Data":"833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe"} Sep 15 21:24:25.237979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:25.237650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:25.271526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:25.271474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" podStartSLOduration=2.310265591 podStartE2EDuration="6.271446274s" totalImagesPullingTime="3.961180683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:25.27112398 +0000 UTC m=+513.495746984" watchObservedRunningTime="2026-09-15 21:24:25.271446274 +0000 UTC m=+513.496069277" Sep 15 21:24:26.933034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:26.933002 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:27.203081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:27.203006 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8bg79" Sep 15 21:24:27.236481 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:27.236430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:27.302332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:27.302307 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:27.507516 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:27.507387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:30.517136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:30.517108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:31.264894 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:31.264849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:31.364965 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:31.364926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:31.465415 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:31.465358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:36.244321 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:36.244290 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:38.098424 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.098386 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t"] Sep 15 21:24:38.105034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.105009 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.123669 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.123625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t"] Sep 15 21:24:38.223299 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.223269 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.223435 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.223334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnkpv\" (UniqueName: \"kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.323693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.323664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnkpv\" (UniqueName: \"kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.323848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.323716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.324093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.324072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.333829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.333808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnkpv\" (UniqueName: \"kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.414880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.414825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:38.695152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.695125 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t"] Sep 15 21:24:38.697152 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:38.697127 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43e4c46_24cf_4c53_a652_33c6871effd7.slice/crio-5cd3ee2028b21aca9d83dc364ff334d012783040e120c5a7e0f4b1197c995a4a WatchSource:0}: Error finding container 5cd3ee2028b21aca9d83dc364ff334d012783040e120c5a7e0f4b1197c995a4a: Status 404 returned error can't find the container with id 5cd3ee2028b21aca9d83dc364ff334d012783040e120c5a7e0f4b1197c995a4a Sep 15 21:24:38.701061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.701034 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt"] Sep 15 21:24:38.701261 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.701239 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" containerName="manager" containerID="cri-o://833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe" gracePeriod=2 Sep 15 21:24:38.709692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.709216 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt"] Sep 15 21:24:38.728011 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.727936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t"] Sep 15 21:24:38.736151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.736118 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:38.736681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.736630 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" containerName="manager" Sep 15 21:24:38.736681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.736657 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" containerName="manager" Sep 15 21:24:38.736866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.736735 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" containerName="manager" Sep 15 21:24:38.742137 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.742115 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t"] Sep 15 21:24:38.742277 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.742212 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.744621 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:38.744590 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:38.755572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.755549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:38.763226 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.763203 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:24:38.766861 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.766832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.783558 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:38.783396 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:38.785532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.785504 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:24:38.827571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.827541 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b2h\" (UniqueName: \"kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.827698 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.827617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgklc\" (UniqueName: \"kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.827698 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.827670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.827820 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.827706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.928888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.928852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.929020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.928959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b2h\" (UniqueName: \"kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.929020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.929015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgklc\" (UniqueName: \"kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.929125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.929065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.929206 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.929186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.929438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.929418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.942717 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.942692 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:38.943123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.943107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b2h\" (UniqueName: \"kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vhvvt\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:38.943565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:38.943549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgklc\" (UniqueName: \"kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-b9fld\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:38.945279 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:38.945234 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.029914 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.029892 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8m6b\" (UniqueName: \"kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b\") pod \"441f2572-ce20-415a-8e06-601d6787e3d0\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " Sep 15 21:24:39.030016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.029926 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume\") pod \"441f2572-ce20-415a-8e06-601d6787e3d0\" (UID: \"441f2572-ce20-415a-8e06-601d6787e3d0\") " Sep 15 21:24:39.030434 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.030415 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "441f2572-ce20-415a-8e06-601d6787e3d0" (UID: "441f2572-ce20-415a-8e06-601d6787e3d0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:39.031773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.031754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b" (OuterVolumeSpecName: "kube-api-access-w8m6b") pod "441f2572-ce20-415a-8e06-601d6787e3d0" (UID: "441f2572-ce20-415a-8e06-601d6787e3d0"). InnerVolumeSpecName "kube-api-access-w8m6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:39.087060 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.087038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:39.094204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.094187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:39.130950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.130911 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8m6b\" (UniqueName: \"kubernetes.io/projected/441f2572-ce20-415a-8e06-601d6787e3d0-kube-api-access-w8m6b\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:39.130950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.130949 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/441f2572-ce20-415a-8e06-601d6787e3d0-extensions-socket-volume\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:39.296405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.296324 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" containerName="manager" containerID="cri-o://4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0" gracePeriod=2 Sep 15 21:24:39.297799 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.297768 2559 generic.go:356] "Generic (PLEG): container finished" podID="441f2572-ce20-415a-8e06-601d6787e3d0" containerID="833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe" exitCode=0 Sep 15 21:24:39.297954 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.297803 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.297954 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.297826 2559 scope.go:117] "RemoveContainer" containerID="833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe" Sep 15 21:24:39.299735 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.299707 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.302176 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.302148 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:39.306498 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.306269 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.306581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.306555 2559 scope.go:117] "RemoveContainer" containerID="833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe" Sep 15 21:24:39.306899 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.306872 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe\": container with ID starting with 833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe not found: ID does not exist" containerID="833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe" Sep 15 21:24:39.306994 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.306908 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe"} err="failed to get container status \"833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe\": rpc error: code = NotFound desc = could not find container \"833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe\": container with ID starting with 833ab7358a8c74027cb2b6ff43139012a06a2aa44163dbb83f4b0a0df6d20dbe not found: ID does not exist" Sep 15 21:24:39.313941 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.313917 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:39.316883 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.316861 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:39.319278 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.319255 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.381078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.381057 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:39.386363 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:39.386338 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a683fbf_d6ed_43f3_aac9_0bcda763f4ce.slice/crio-cd30b7679924c1fbbe631155cbab24c0e841e6df5b858572fbfb74be97d2e689 WatchSource:0}: Error finding container cd30b7679924c1fbbe631155cbab24c0e841e6df5b858572fbfb74be97d2e689: Status 404 returned error can't find the container with id cd30b7679924c1fbbe631155cbab24c0e841e6df5b858572fbfb74be97d2e689 Sep 15 21:24:39.393609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.393585 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:24:39.396787 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:39.396763 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18a9451_23fb_4c20_b92c_29454a0c1f6e.slice/crio-42e92b1cc3d0ff59ccef655c8e3292e425863a3ca7934479adbee3314c49a5de WatchSource:0}: Error finding container 42e92b1cc3d0ff59ccef655c8e3292e425863a3ca7934479adbee3314c49a5de: Status 404 returned error can't find the container with id 42e92b1cc3d0ff59ccef655c8e3292e425863a3ca7934479adbee3314c49a5de Sep 15 21:24:39.529655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.529632 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:39.531998 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.531974 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vvzrt\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vvzrt" Sep 15 21:24:39.534129 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:39.534105 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:39.635422 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.635393 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume\") pod \"d43e4c46-24cf-4c53-a652-33c6871effd7\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " Sep 15 21:24:39.635586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.635470 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnkpv\" (UniqueName: \"kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv\") pod \"d43e4c46-24cf-4c53-a652-33c6871effd7\" (UID: \"d43e4c46-24cf-4c53-a652-33c6871effd7\") " Sep 15 21:24:39.635648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.635624 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d43e4c46-24cf-4c53-a652-33c6871effd7" (UID: "d43e4c46-24cf-4c53-a652-33c6871effd7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:39.637401 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.637380 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv" (OuterVolumeSpecName: "kube-api-access-rnkpv") pod "d43e4c46-24cf-4c53-a652-33c6871effd7" (UID: "d43e4c46-24cf-4c53-a652-33c6871effd7"). InnerVolumeSpecName "kube-api-access-rnkpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:39.736125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.736093 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d43e4c46-24cf-4c53-a652-33c6871effd7-extensions-socket-volume\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:39.736125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:39.736119 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnkpv\" (UniqueName: \"kubernetes.io/projected/d43e4c46-24cf-4c53-a652-33c6871effd7-kube-api-access-rnkpv\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:40.167176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.167119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:40.281284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.281244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:40.290227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.290196 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441f2572-ce20-415a-8e06-601d6787e3d0" path="/var/lib/kubelet/pods/441f2572-ce20-415a-8e06-601d6787e3d0/volumes" Sep 15 21:24:40.290573 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.290560 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" path="/var/lib/kubelet/pods/d43e4c46-24cf-4c53-a652-33c6871effd7/volumes" Sep 15 21:24:40.303171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.303145 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" event={"ID":"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce","Type":"ContainerStarted","Data":"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b"} Sep 15 21:24:40.303272 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.303190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" event={"ID":"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce","Type":"ContainerStarted","Data":"cd30b7679924c1fbbe631155cbab24c0e841e6df5b858572fbfb74be97d2e689"} Sep 15 21:24:40.303333 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.303268 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:40.304521 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.304502 2559 generic.go:356] "Generic (PLEG): container finished" podID="d43e4c46-24cf-4c53-a652-33c6871effd7" containerID="4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0" exitCode=2 Sep 15 21:24:40.304612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.304573 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:40.304612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.304587 2559 scope.go:117] "RemoveContainer" containerID="4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0" Sep 15 21:24:40.306632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.306604 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" event={"ID":"c18a9451-23fb-4c20-b92c-29454a0c1f6e","Type":"ContainerStarted","Data":"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205"} Sep 15 21:24:40.306738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.306647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" event={"ID":"c18a9451-23fb-4c20-b92c-29454a0c1f6e","Type":"ContainerStarted","Data":"42e92b1cc3d0ff59ccef655c8e3292e425863a3ca7934479adbee3314c49a5de"} Sep 15 21:24:40.306738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.306699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:40.312978 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.312961 2559 scope.go:117] "RemoveContainer" containerID="4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0" Sep 15 21:24:40.313235 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:40.313216 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0\": container with ID starting with 4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0 not found: ID does not exist" containerID="4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0" Sep 15 21:24:40.313271 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.313243 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0"} err="failed to get container status \"4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0\": rpc error: code = NotFound desc = could not find container \"4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0\": container with ID starting with 4d398c95aa1dc7c1124ef68f6efb99e78dd67ba5a955dafbff770cf91f24d1e0 not found: ID does not exist" Sep 15 21:24:40.336360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.336318 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" podStartSLOduration=2.336305708 podStartE2EDuration="2.336305708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:40.335095476 +0000 UTC m=+528.559718480" watchObservedRunningTime="2026-09-15 21:24:40.336305708 +0000 UTC m=+528.560928709" Sep 15 21:24:40.370699 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:40.370670 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:40.371426 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:40.371383 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" podStartSLOduration=2.371365504 podStartE2EDuration="2.371365504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:40.367563127 +0000 UTC m=+528.592186132" watchObservedRunningTime="2026-09-15 21:24:40.371365504 +0000 UTC m=+528.595988508" Sep 15 21:24:42.288744 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:42.288708 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-qt45t\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qt45t" Sep 15 21:24:42.738360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:42.738326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:42.888675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:42.888641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:45.909185 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:45.909098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:46.011852 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:46.011810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:50.512428 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:50.512397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:50.610650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:50.610619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:50.711618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:50.711586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:50.812418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:50.812346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:50.991796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:50.991764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:51.113657 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.113634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:51.315955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.315931 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:24:51.316107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.316094 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:51.454433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.454360 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:51.454599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.454558 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" podUID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" containerName="manager" containerID="cri-o://c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b" gracePeriod=10 Sep 15 21:24:51.689092 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.689070 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:51.725931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.725876 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgklc\" (UniqueName: \"kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc\") pod \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " Sep 15 21:24:51.725931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.725914 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume\") pod \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\" (UID: \"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce\") " Sep 15 21:24:51.726302 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.726278 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" (UID: "7a683fbf-d6ed-43f3-aac9-0bcda763f4ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:24:51.727826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.727799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc" (OuterVolumeSpecName: "kube-api-access-fgklc") pod "7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" (UID: "7a683fbf-d6ed-43f3-aac9-0bcda763f4ce"). InnerVolumeSpecName "kube-api-access-fgklc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:24:51.826561 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.826537 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgklc\" (UniqueName: \"kubernetes.io/projected/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-kube-api-access-fgklc\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:51.826561 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.826557 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce-extensions-socket-volume\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:24:51.976179 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976111 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:24:51.976496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976481 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" containerName="manager" Sep 15 21:24:51.976540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976498 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" containerName="manager" Sep 15 21:24:51.976540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976508 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" containerName="manager" Sep 15 21:24:51.976540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976513 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" containerName="manager" Sep 15 21:24:51.976626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976564 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43e4c46-24cf-4c53-a652-33c6871effd7" containerName="manager" Sep 15 21:24:51.976626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.976572 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" containerName="manager" Sep 15 21:24:51.979495 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:51.979481 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.005218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.005188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:24:52.027893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.027869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.027991 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.027937 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bknkg\" (UniqueName: \"kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.128383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.128343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bknkg\" (UniqueName: \"kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.128525 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.128400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.128778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.128760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.142845 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.142820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknkg\" (UniqueName: \"kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n8289\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.288678 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.288640 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:52.361635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.361605 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" containerID="c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b" exitCode=0 Sep 15 21:24:52.361791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.361641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" event={"ID":"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce","Type":"ContainerDied","Data":"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b"} Sep 15 21:24:52.361791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.361671 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" Sep 15 21:24:52.361791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.361712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld" event={"ID":"7a683fbf-d6ed-43f3-aac9-0bcda763f4ce","Type":"ContainerDied","Data":"cd30b7679924c1fbbe631155cbab24c0e841e6df5b858572fbfb74be97d2e689"} Sep 15 21:24:52.361791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.361744 2559 scope.go:117] "RemoveContainer" containerID="c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b" Sep 15 21:24:52.370995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.370978 2559 scope.go:117] "RemoveContainer" containerID="c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b" Sep 15 21:24:52.371301 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:24:52.371280 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b\": container with ID starting with c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b not found: ID does not exist" containerID="c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b" Sep 15 21:24:52.371399 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.371309 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b"} err="failed to get container status \"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b\": rpc error: code = NotFound desc = could not find container \"c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b\": container with ID starting with c10a3a63873bcf36b923b8d8db6e3794e006ebd972a0664fb45ac06901b5087b not found: ID does not exist" Sep 15 21:24:52.388045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.388018 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:52.394634 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.394615 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-b9fld"] Sep 15 21:24:52.569923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:52.569898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:24:52.572286 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:24:52.572245 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484d69e4_1c76_4088_bca6_fedb65acaa23.slice/crio-c8da05744cf6233a4854d0a8030180bcde827d3141fa119fdac243374861a6d6 WatchSource:0}: Error finding container c8da05744cf6233a4854d0a8030180bcde827d3141fa119fdac243374861a6d6: Status 404 returned error can't find the container with id c8da05744cf6233a4854d0a8030180bcde827d3141fa119fdac243374861a6d6 Sep 15 21:24:53.369781 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:53.369740 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerStarted","Data":"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314"} Sep 15 21:24:53.370175 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:53.369845 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:24:53.370175 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:53.369878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerStarted","Data":"c8da05744cf6233a4854d0a8030180bcde827d3141fa119fdac243374861a6d6"} Sep 15 21:24:53.392591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:53.392282 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" podStartSLOduration=2.392262312 podStartE2EDuration="2.392262312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:53.390857476 +0000 UTC m=+541.615480480" watchObservedRunningTime="2026-09-15 21:24:53.392262312 +0000 UTC m=+541.616885317" Sep 15 21:24:54.289578 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:54.289545 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a683fbf-d6ed-43f3-aac9-0bcda763f4ce" path="/var/lib/kubelet/pods/7a683fbf-d6ed-43f3-aac9-0bcda763f4ce/volumes" Sep 15 21:24:59.299030 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:59.298990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:24:59.398685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:24:59.398654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:04.376579 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.376546 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:25:04.429892 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.429858 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:25:04.430070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.430044 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" podUID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" containerName="manager" containerID="cri-o://ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205" gracePeriod=10 Sep 15 21:25:04.667505 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.667475 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:25:04.728637 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.728610 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85b2h\" (UniqueName: \"kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h\") pod \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " Sep 15 21:25:04.728770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.728686 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume\") pod \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\" (UID: \"c18a9451-23fb-4c20-b92c-29454a0c1f6e\") " Sep 15 21:25:04.729034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.729007 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c18a9451-23fb-4c20-b92c-29454a0c1f6e" (UID: "c18a9451-23fb-4c20-b92c-29454a0c1f6e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:25:04.730664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.730641 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h" (OuterVolumeSpecName: "kube-api-access-85b2h") pod "c18a9451-23fb-4c20-b92c-29454a0c1f6e" (UID: "c18a9451-23fb-4c20-b92c-29454a0c1f6e"). InnerVolumeSpecName "kube-api-access-85b2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:25:04.829771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.829741 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b2h\" (UniqueName: \"kubernetes.io/projected/c18a9451-23fb-4c20-b92c-29454a0c1f6e-kube-api-access-85b2h\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:25:04.829771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:04.829766 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18a9451-23fb-4c20-b92c-29454a0c1f6e-extensions-socket-volume\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:25:05.421924 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.421890 2559 generic.go:356] "Generic (PLEG): container finished" podID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" containerID="ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205" exitCode=0 Sep 15 21:25:05.422380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.421933 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" Sep 15 21:25:05.422380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.421933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" event={"ID":"c18a9451-23fb-4c20-b92c-29454a0c1f6e","Type":"ContainerDied","Data":"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205"} Sep 15 21:25:05.422380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.422054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt" event={"ID":"c18a9451-23fb-4c20-b92c-29454a0c1f6e","Type":"ContainerDied","Data":"42e92b1cc3d0ff59ccef655c8e3292e425863a3ca7934479adbee3314c49a5de"} Sep 15 21:25:05.422380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.422077 2559 scope.go:117] "RemoveContainer" containerID="ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205" Sep 15 21:25:05.430141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.430123 2559 scope.go:117] "RemoveContainer" containerID="ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205" Sep 15 21:25:05.430470 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:25:05.430435 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205\": container with ID starting with ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205 not found: ID does not exist" containerID="ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205" Sep 15 21:25:05.430519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.430481 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205"} err="failed to get container status \"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205\": rpc error: code = NotFound desc = could not find container \"ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205\": container with ID starting with ea8ec4cad02e5fccce0399a0052d361dcc18156517fba09ff84493abbae8a205 not found: ID does not exist" Sep 15 21:25:05.443177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.443152 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:25:05.449427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:05.449406 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vhvvt"] Sep 15 21:25:06.289184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:06.289143 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" path="/var/lib/kubelet/pods/c18a9451-23fb-4c20-b92c-29454a0c1f6e/volumes" Sep 15 21:25:07.829363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.829331 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps"] Sep 15 21:25:07.829743 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.829668 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" containerName="manager" Sep 15 21:25:07.829743 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.829681 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" containerName="manager" Sep 15 21:25:07.829743 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.829733 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18a9451-23fb-4c20-b92c-29454a0c1f6e" containerName="manager" Sep 15 21:25:07.833897 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.833879 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.836855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.836836 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-44qsb\"" Sep 15 21:25:07.849472 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.849432 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps"] Sep 15 21:25:07.867825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.867800 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:07.954114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05505b04-0c94-467f-9e72-bc3af758b0c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qjk\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-kube-api-access-97qjk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954261 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.954341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.954333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:07.962291 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:07.962269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:08.055132 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05505b04-0c94-467f-9e72-bc3af758b0c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055503 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qjk\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-kube-api-access-97qjk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.055983 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.055898 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.056136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.056045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.056336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.056314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05505b04-0c94-467f-9e72-bc3af758b0c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.057571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.057552 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.058040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.058022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.063323 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.063301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.063569 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.063551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qjk\" (UniqueName: \"kubernetes.io/projected/05505b04-0c94-467f-9e72-bc3af758b0c2-kube-api-access-97qjk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-574ps\" (UID: \"05505b04-0c94-467f-9e72-bc3af758b0c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.143205 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.143141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:08.418919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.418857 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps"] Sep 15 21:25:08.422091 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:25:08.422063 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05505b04_0c94_467f_9e72_bc3af758b0c2.slice/crio-f54ad0b0246835ca91f646d64a02864bbdeab7bb28423b36399a1085bb664d12 WatchSource:0}: Error finding container f54ad0b0246835ca91f646d64a02864bbdeab7bb28423b36399a1085bb664d12: Status 404 returned error can't find the container with id f54ad0b0246835ca91f646d64a02864bbdeab7bb28423b36399a1085bb664d12 Sep 15 21:25:08.424206 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.424173 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:25:08.424298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.424238 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:25:08.424298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.424268 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:25:08.437105 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:08.437079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" event={"ID":"05505b04-0c94-467f-9e72-bc3af758b0c2","Type":"ContainerStarted","Data":"f54ad0b0246835ca91f646d64a02864bbdeab7bb28423b36399a1085bb664d12"} Sep 15 21:25:09.443842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:09.443798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" event={"ID":"05505b04-0c94-467f-9e72-bc3af758b0c2","Type":"ContainerStarted","Data":"6ca78fafe7e8e906d9011b3a406c5b2f0df8d2c2bf0ba4e880a98b3e3db45d7c"} Sep 15 21:25:09.467692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:09.467648 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podStartSLOduration=2.467633558 podStartE2EDuration="2.467633558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:09.465646406 +0000 UTC m=+557.690269434" watchObservedRunningTime="2026-09-15 21:25:09.467633558 +0000 UTC m=+557.692256563" Sep 15 21:25:10.143942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:10.143908 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:10.148606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:10.148577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:10.448563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:10.448488 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:10.449375 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:10.449357 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:25:24.329324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.329278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:24.427427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.427397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:24.487916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.487887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:24.721744 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.721661 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:25:24.724669 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.724647 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.727339 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.727318 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 15 21:25:24.727488 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.727373 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q9527\"" Sep 15 21:25:24.736407 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.736382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:25:24.749072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.749043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:25:24.887070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.887038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hts88\" (UniqueName: \"kubernetes.io/projected/d40035ef-9cb5-46dd-9167-ef5113a68871-kube-api-access-hts88\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.887186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.887133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d40035ef-9cb5-46dd-9167-ef5113a68871-config-file\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.988258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.988187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d40035ef-9cb5-46dd-9167-ef5113a68871-config-file\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.988371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.988287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hts88\" (UniqueName: \"kubernetes.io/projected/d40035ef-9cb5-46dd-9167-ef5113a68871-kube-api-access-hts88\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.988905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.988886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d40035ef-9cb5-46dd-9167-ef5113a68871-config-file\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:24.996826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:24.996808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hts88\" (UniqueName: \"kubernetes.io/projected/d40035ef-9cb5-46dd-9167-ef5113a68871-kube-api-access-hts88\") pod \"limitador-limitador-84994f7ffc-gbtkd\" (UID: \"d40035ef-9cb5-46dd-9167-ef5113a68871\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:25.035357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:25.035332 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:25.195625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:25.195590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:25.295630 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:25.295557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:25.302966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:25.302946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:25:25.304985 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:25:25.304960 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40035ef_9cb5_46dd_9167_ef5113a68871.slice/crio-b6a17f52d05e9d561695cc380e354096ff6dde799f22e618d29703dbd02ad5c8 WatchSource:0}: Error finding container b6a17f52d05e9d561695cc380e354096ff6dde799f22e618d29703dbd02ad5c8: Status 404 returned error can't find the container with id b6a17f52d05e9d561695cc380e354096ff6dde799f22e618d29703dbd02ad5c8 Sep 15 21:25:25.516133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:25.516100 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" event={"ID":"d40035ef-9cb5-46dd-9167-ef5113a68871","Type":"ContainerStarted","Data":"b6a17f52d05e9d561695cc380e354096ff6dde799f22e618d29703dbd02ad5c8"} Sep 15 21:25:27.527464 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:27.527414 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" event={"ID":"d40035ef-9cb5-46dd-9167-ef5113a68871","Type":"ContainerStarted","Data":"d1f44b37010b28bb8339afa44d0fbdebf9f63524c46189131611724c0fbd3d5d"} Sep 15 21:25:27.527871 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:27.527489 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:27.546111 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:27.546064 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" podStartSLOduration=2.152168745 podStartE2EDuration="3.546050637s" totalImagesPullingTime="1.393881892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:27.544130317 +0000 UTC m=+575.768753321" watchObservedRunningTime="2026-09-15 21:25:27.546050637 +0000 UTC m=+575.770673641" Sep 15 21:25:38.533255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:38.533219 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gbtkd" Sep 15 21:25:41.921076 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:41.921030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:42.017129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:42.017078 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:42.366332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:42.366280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:54.837621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:54.837578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:25:55.079873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:25:55.079842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:01.512019 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:01.511989 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:01.611768 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:01.611729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:01.700157 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:01.700113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:01.793962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:01.793875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:01.852314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:01.852286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:05.226089 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:05.226057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:05.455789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:05.455758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:07.198187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:07.198155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:38.040294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:38.040259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:39.600466 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:39.600399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:39.855838 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:39.855758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:40.056499 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:40.056443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:40.228508 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:40.228408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:40.444769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:40.444735 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:26:40.464023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:40.463981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:49.889308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.889247 2559 generic.go:356] "Generic (PLEG): container finished" podID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerID="6ca78fafe7e8e906d9011b3a406c5b2f0df8d2c2bf0ba4e880a98b3e3db45d7c" exitCode=0 Sep 15 21:26:49.889689 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.889320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" event={"ID":"05505b04-0c94-467f-9e72-bc3af758b0c2","Type":"ContainerDied","Data":"6ca78fafe7e8e906d9011b3a406c5b2f0df8d2c2bf0ba4e880a98b3e3db45d7c"} Sep 15 21:26:49.889689 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.889681 2559 scope.go:117] "RemoveContainer" containerID="6ca78fafe7e8e906d9011b3a406c5b2f0df8d2c2bf0ba4e880a98b3e3db45d7c" Sep 15 21:26:49.890769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.890738 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:26:49.890839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.890822 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:26:49.890884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:49.890868 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:26:50.896955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:50.896921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" event={"ID":"05505b04-0c94-467f-9e72-bc3af758b0c2","Type":"ContainerStarted","Data":"1b4ea3f704cf629f91f983f34d094e29da2f1eb7696f5998af69ed886f35d0e8"} Sep 15 21:26:51.144007 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:51.143980 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:26:51.144729 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:51.144707 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-574ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Sep 15 21:26:51.144811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:51.144752 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podUID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Sep 15 21:26:52.087271 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:52.087228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:52.143738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:52.143708 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-574ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Sep 15 21:26:52.143879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:52.143762 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podUID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Sep 15 21:26:52.181141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:52.181055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:26:53.143704 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:53.143663 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-574ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Sep 15 21:26:53.144083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:53.143724 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podUID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Sep 15 21:26:54.143906 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:54.143872 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-574ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Sep 15 21:26:54.144262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:54.143925 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podUID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Sep 15 21:26:55.144355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:55.144316 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-574ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Sep 15 21:26:55.144804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:55.144775 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" podUID="05505b04-0c94-467f-9e72-bc3af758b0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Sep 15 21:26:55.449443 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:55.449369 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:26:56.147010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:56.146978 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:26:56.147825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:26:56.147806 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-574ps" Sep 15 21:27:04.056662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:04.056632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:04.458048 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:04.458009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:10.243326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:10.243295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:10.933373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:10.933338 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:13.409288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.409256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:13.514713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.514683 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp"] Sep 15 21:27:13.518406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.518383 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.521245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.521222 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 21:27:13.521349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.521233 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 21:27:13.521349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.521249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fxwc6\"" Sep 15 21:27:13.522322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.522304 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 15 21:27:13.529738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.529717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp"] Sep 15 21:27:13.549588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.549561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:13.636080 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.636237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bc4a20-3914-4fa3-893c-161a4bb87192-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.636237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6mh\" (UniqueName: \"kubernetes.io/projected/e7bc4a20-3914-4fa3-893c-161a4bb87192-kube-api-access-dj6mh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.636237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.636237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.636418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.636297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737420 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bc4a20-3914-4fa3-893c-161a4bb87192-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737420 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6mh\" (UniqueName: \"kubernetes.io/projected/e7bc4a20-3914-4fa3-893c-161a4bb87192-kube-api-access-dj6mh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.737909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.738005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.737957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.738051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.738030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.739742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.739722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7bc4a20-3914-4fa3-893c-161a4bb87192-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.739830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.739812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7bc4a20-3914-4fa3-893c-161a4bb87192-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.747704 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.747685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6mh\" (UniqueName: \"kubernetes.io/projected/e7bc4a20-3914-4fa3-893c-161a4bb87192-kube-api-access-dj6mh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp\" (UID: \"e7bc4a20-3914-4fa3-893c-161a4bb87192\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:13.830390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:13.830358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:14.107833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:14.107805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp"] Sep 15 21:27:14.109805 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:27:14.109778 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bc4a20_3914_4fa3_893c_161a4bb87192.slice/crio-3e42e5973d9fc40609c242591a16493899c15e8d647bcb89a9dddff3e53edd09 WatchSource:0}: Error finding container 3e42e5973d9fc40609c242591a16493899c15e8d647bcb89a9dddff3e53edd09: Status 404 returned error can't find the container with id 3e42e5973d9fc40609c242591a16493899c15e8d647bcb89a9dddff3e53edd09 Sep 15 21:27:14.111067 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:14.111049 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:27:14.713851 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:14.713816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:15.002484 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:15.002381 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" event={"ID":"e7bc4a20-3914-4fa3-893c-161a4bb87192","Type":"ContainerStarted","Data":"3e42e5973d9fc40609c242591a16493899c15e8d647bcb89a9dddff3e53edd09"} Sep 15 21:27:16.008860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:16.008826 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" event={"ID":"e7bc4a20-3914-4fa3-893c-161a4bb87192","Type":"ContainerStarted","Data":"fd56123c1889ca7b5abe1433837dbf6ddd851fc9b74b1d4e3e654be8a5c303b8"} Sep 15 21:27:16.029003 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:16.028958 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" podStartSLOduration=2.023986773 podStartE2EDuration="3.028943623s" totalImagesPullingTime="1.00495685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:16.02749235 +0000 UTC m=+684.252115356" watchObservedRunningTime="2026-09-15 21:27:16.028943623 +0000 UTC m=+684.253566627" Sep 15 21:27:19.045967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:19.045933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:20.138730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:20.138698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:23.146258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:23.146223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:23.247040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:23.247007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:23.830897 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:23.830866 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:23.831043 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:23.830928 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:23.843200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:23.843178 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:24.055279 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:24.055247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp" Sep 15 21:27:24.546899 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:24.546869 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:26.017599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:26.017569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:31.128050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.128017 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs"] Sep 15 21:27:31.131425 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.131404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.134071 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.134052 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 15 21:27:31.140626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.140605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs"] Sep 15 21:27:31.151740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.151710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:31.295090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff865\" (UniqueName: \"kubernetes.io/projected/5c45d377-c596-4aee-9fcf-31014767d8a8-kube-api-access-ff865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.295255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295095 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.295255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.295351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.295351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.295351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.295328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c45d377-c596-4aee-9fcf-31014767d8a8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396409 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396409 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c45d377-c596-4aee-9fcf-31014767d8a8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396409 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff865\" (UniqueName: \"kubernetes.io/projected/5c45d377-c596-4aee-9fcf-31014767d8a8-kube-api-access-ff865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.396907 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.397037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.396959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.397037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.397006 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.398688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.398668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c45d377-c596-4aee-9fcf-31014767d8a8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.398964 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.398945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c45d377-c596-4aee-9fcf-31014767d8a8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.405447 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.405428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff865\" (UniqueName: \"kubernetes.io/projected/5c45d377-c596-4aee-9fcf-31014767d8a8-kube-api-access-ff865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f7hcs\" (UID: \"5c45d377-c596-4aee-9fcf-31014767d8a8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.441945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.441923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:31.714990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:31.714968 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs"] Sep 15 21:27:31.717229 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:27:31.717197 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c45d377_c596_4aee_9fcf_31014767d8a8.slice/crio-b30260183de9c1bfbbf85ec7f103453de9a5bbcfb3fe6c9e03576a7e39125327 WatchSource:0}: Error finding container b30260183de9c1bfbbf85ec7f103453de9a5bbcfb3fe6c9e03576a7e39125327: Status 404 returned error can't find the container with id b30260183de9c1bfbbf85ec7f103453de9a5bbcfb3fe6c9e03576a7e39125327 Sep 15 21:27:32.083832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:32.083799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" event={"ID":"5c45d377-c596-4aee-9fcf-31014767d8a8","Type":"ContainerStarted","Data":"eebf68f8e4c19e4e966c6f63deb8a8cd3a7a6fe14d62e3ea8b3ed86ce01396a5"} Sep 15 21:27:32.083974 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:32.083849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" event={"ID":"5c45d377-c596-4aee-9fcf-31014767d8a8","Type":"ContainerStarted","Data":"b30260183de9c1bfbbf85ec7f103453de9a5bbcfb3fe6c9e03576a7e39125327"} Sep 15 21:27:32.104679 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:32.104638 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" podStartSLOduration=0.935509233 podStartE2EDuration="1.104623526s" totalImagesPullingTime="169.114293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:32.102766794 +0000 UTC m=+700.327389892" watchObservedRunningTime="2026-09-15 21:27:32.104623526 +0000 UTC m=+700.329246527" Sep 15 21:27:33.122873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:33.122835 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:41.442769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:41.442736 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:41.443154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:41.442787 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:41.454612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:41.454589 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:42.027199 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.027170 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj"] Sep 15 21:27:42.030594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.030577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.033284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.033265 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 15 21:27:42.040755 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.040734 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj"] Sep 15 21:27:42.144011 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.143977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f7hcs" Sep 15 21:27:42.148747 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.148717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:42.182728 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce248c16-7ecd-41ec-b506-42724441778e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.182888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.182888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.183006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8q5\" (UniqueName: \"kubernetes.io/projected/ce248c16-7ecd-41ec-b506-42724441778e-kube-api-access-6z8q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.183006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.183006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.182999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283528 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283528 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8q5\" (UniqueName: \"kubernetes.io/projected/ce248c16-7ecd-41ec-b506-42724441778e-kube-api-access-6z8q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283750 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283750 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283750 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce248c16-7ecd-41ec-b506-42724441778e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.283750 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.283650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.284039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.284012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.284398 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.284369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.284625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.284600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.287567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.287533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce248c16-7ecd-41ec-b506-42724441778e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.289804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.289774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce248c16-7ecd-41ec-b506-42724441778e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.297162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.294175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8q5\" (UniqueName: \"kubernetes.io/projected/ce248c16-7ecd-41ec-b506-42724441778e-kube-api-access-6z8q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj\" (UID: \"ce248c16-7ecd-41ec-b506-42724441778e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.339946 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.339912 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:42.612535 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:42.612513 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj"] Sep 15 21:27:42.615041 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:27:42.615010 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce248c16_7ecd_41ec_b506_42724441778e.slice/crio-04b73aaf62d842ca4407c37e65d4d278f86bd4cef56919d9ee065257cc143118 WatchSource:0}: Error finding container 04b73aaf62d842ca4407c37e65d4d278f86bd4cef56919d9ee065257cc143118: Status 404 returned error can't find the container with id 04b73aaf62d842ca4407c37e65d4d278f86bd4cef56919d9ee065257cc143118 Sep 15 21:27:43.138240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:43.138196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" event={"ID":"ce248c16-7ecd-41ec-b506-42724441778e","Type":"ContainerStarted","Data":"7f9961d38971e9db33bc0907893e2012e453bee7751e09a2708d60017b4c9444"} Sep 15 21:27:43.138414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:43.138250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" event={"ID":"ce248c16-7ecd-41ec-b506-42724441778e","Type":"ContainerStarted","Data":"04b73aaf62d842ca4407c37e65d4d278f86bd4cef56919d9ee065257cc143118"} Sep 15 21:27:43.159036 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:43.158992 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" podStartSLOduration=0.98056257 podStartE2EDuration="1.158978225s" totalImagesPullingTime="178.415655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:43.157674802 +0000 UTC m=+711.382297819" watchObservedRunningTime="2026-09-15 21:27:43.158978225 +0000 UTC m=+711.383601229" Sep 15 21:27:44.433678 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:44.433642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:27:49.859650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:49.859566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:49.960336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:49.960301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:27:52.340819 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:52.340790 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:52.341231 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:52.340837 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:52.353091 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:52.353062 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:27:53.198958 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:27:53.198925 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj" Sep 15 21:28:58.292505 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:28:58.292444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:28:58.374242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:28:58.374207 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:28:59.398664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:28:59.398633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:28:59.539501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:28:59.539467 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.057987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.057911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.241227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.241191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.456680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.456649 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:29:15.459808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.459782 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.462914 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.462892 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b99d7b-openshift-default-dockercfg-wgfbk\"" Sep 15 21:29:15.488237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.488211 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:29:15.542769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.542716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.576595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxb64\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.576929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.577079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.576929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.645628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.645405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.657062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.657031 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:29:15.661305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.661280 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.665200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.665182 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4bc420-openshift-default-dockercfg-p6vlk\"" Sep 15 21:29:15.678273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678229 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:29:15.678273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxb64\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678583 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678671 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678782 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.678869 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678843 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.678905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.679273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.679389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679516 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.679288 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679585 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.679531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.679686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.679657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.683494 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.682217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.684042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.684021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.690554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.690521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.691617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.691595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxb64\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64\") pod \"e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.737716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.737633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.771900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.771705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:15.780563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780536 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780689 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780689 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780775 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780775 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ch45\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.780928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.780881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.838905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.838834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.881658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.881804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.881804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.881804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ch45\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.881804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.881944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.882024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.882138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882598 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.882576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.882616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.882996 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.882974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.884016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.883921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.884760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.884738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.890931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.890896 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.891342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.891324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ch45\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45\") pod \"e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:15.937915 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.937891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:15.973117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:15.973084 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:16.040793 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.040764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:16.064128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.064106 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:29:16.065824 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:16.065795 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b8361_c00d_4104_86f9_651ce8b839da.slice/crio-4b7b1a227fd0202ccb0a6eb4b8d303e457b1fe2f39db744a259325bda8bf663e WatchSource:0}: Error finding container 4b7b1a227fd0202ccb0a6eb4b8d303e457b1fe2f39db744a259325bda8bf663e: Status 404 returned error can't find the container with id 4b7b1a227fd0202ccb0a6eb4b8d303e457b1fe2f39db744a259325bda8bf663e Sep 15 21:29:16.067991 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.067960 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.068055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.068022 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.068055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.068050 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.248426 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.248395 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:29:16.250952 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:16.250925 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d53c5aa_c553_4c19_9437_cd27274ff201.slice/crio-eeeb99f7d3de8a1fc91fe8c0245dd40001648884655a0c58f2eeb01e1bf649d3 WatchSource:0}: Error finding container eeeb99f7d3de8a1fc91fe8c0245dd40001648884655a0c58f2eeb01e1bf649d3: Status 404 returned error can't find the container with id eeeb99f7d3de8a1fc91fe8c0245dd40001648884655a0c58f2eeb01e1bf649d3 Sep 15 21:29:16.253255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.253221 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.253365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.253286 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.253365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.253314 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:16.414489 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.414424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:16.511394 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.511357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:16.553850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.553758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerStarted","Data":"a9bf09c12280b109f84160dcd7c784072aa974e60bb73727d3591fe66c794edd"} Sep 15 21:29:16.553850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.553818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerStarted","Data":"4b7b1a227fd0202ccb0a6eb4b8d303e457b1fe2f39db744a259325bda8bf663e"} Sep 15 21:29:16.563732 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.563700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerStarted","Data":"febdcedd8fef371fb2a2808a99c7fc99c20b7c26352fd67b081970c3c6a41006"} Sep 15 21:29:16.563958 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.563936 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerStarted","Data":"eeeb99f7d3de8a1fc91fe8c0245dd40001648884655a0c58f2eeb01e1bf649d3"} Sep 15 21:29:16.583758 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.583709 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podStartSLOduration=1.58369539 podStartE2EDuration="1.58369539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:16.581420882 +0000 UTC m=+804.806043897" watchObservedRunningTime="2026-09-15 21:29:16.58369539 +0000 UTC m=+804.808318394" Sep 15 21:29:16.772001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.771965 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:16.840169 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.840140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:16.973778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:16.973740 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:17.777893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:17.777859 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:17.800732 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:17.800682 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podStartSLOduration=2.800661627 podStartE2EDuration="2.800661627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:16.608407288 +0000 UTC m=+804.833030292" watchObservedRunningTime="2026-09-15 21:29:17.800661627 +0000 UTC m=+806.025284632" Sep 15 21:29:17.978499 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:17.978467 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:18.523350 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.523318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:18.573968 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.573932 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:18.574134 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.573993 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:18.574660 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.574639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:18.574778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.574694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:18.610429 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.610401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:18.717328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.717299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:18.890448 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:18.890417 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:19.230760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:19.230690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:29:19.786345 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:19.786309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:19.883485 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:19.883424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:20.092968 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:20.092924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:20.439305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:20.439181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:20.587143 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:20.587114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:21.078588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.078553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:21.179363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.179331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:21.278793 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.278767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:21.775504 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.775464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7763-917d36"] Sep 15 21:29:21.781825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.781791 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:21.785120 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.785098 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\"" Sep 15 21:29:21.793979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.793949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7763-917d36"] Sep 15 21:29:21.935085 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:21.935048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2wg\" (UniqueName: \"kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg\") pod \"test-curl-7763-917d36\" (UID: \"b59aca09-5ed6-42de-b753-444cac5584ba\") " pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:22.035882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.035809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2wg\" (UniqueName: \"kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg\") pod \"test-curl-7763-917d36\" (UID: \"b59aca09-5ed6-42de-b753-444cac5584ba\") " pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:22.064093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.064066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2wg\" (UniqueName: \"kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg\") pod \"test-curl-7763-917d36\" (UID: \"b59aca09-5ed6-42de-b753-444cac5584ba\") " pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:22.092045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.092011 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:22.192955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.192921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:22.293147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.293066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:22.365049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.365025 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7763-917d36"] Sep 15 21:29:22.368197 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:22.368168 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59aca09_5ed6_42de_b753_444cac5584ba.slice/crio-7d1401d9a28d0e2f247f3f6dcbe1ae70747ffaa946e10c834b55dc9dac52ee2f WatchSource:0}: Error finding container 7d1401d9a28d0e2f247f3f6dcbe1ae70747ffaa946e10c834b55dc9dac52ee2f: Status 404 returned error can't find the container with id 7d1401d9a28d0e2f247f3f6dcbe1ae70747ffaa946e10c834b55dc9dac52ee2f Sep 15 21:29:22.595598 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.595563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7763-917d36" event={"ID":"b59aca09-5ed6-42de-b753-444cac5584ba","Type":"ContainerStarted","Data":"7d1401d9a28d0e2f247f3f6dcbe1ae70747ffaa946e10c834b55dc9dac52ee2f"} Sep 15 21:29:22.697233 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.697200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:22.795919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:22.795891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:23.183676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:23.183648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:23.282996 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:23.282965 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:23.602426 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:23.602397 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7763-917d36" event={"ID":"b59aca09-5ed6-42de-b753-444cac5584ba","Type":"ContainerStarted","Data":"9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd"} Sep 15 21:29:23.616853 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:23.616808 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7763-917d36" podStartSLOduration=1.8421416659999998 podStartE2EDuration="2.616792917s" totalImagesPullingTime="774.651251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:23.61582619 +0000 UTC m=+811.840449194" watchObservedRunningTime="2026-09-15 21:29:23.616792917 +0000 UTC m=+811.841415921" Sep 15 21:29:23.773156 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:23.773123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.123856 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.123823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.204877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.204847 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7763-917d36"] Sep 15 21:29:24.208631 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.208589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.209369 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.209345 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7763-917d36"] Sep 15 21:29:24.364802 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.364761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.607353 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:29:24.607317 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7763-917d36" container="test-curl-7763-917d36" Sep 15 21:29:24.638296 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.638223 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:29:24.638623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.638605 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59aca09-5ed6-42de-b753-444cac5584ba" containerName="test-curl-7763-917d36" Sep 15 21:29:24.638691 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.638624 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59aca09-5ed6-42de-b753-444cac5584ba" containerName="test-curl-7763-917d36" Sep 15 21:29:24.638748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.638694 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59aca09-5ed6-42de-b753-444cac5584ba" containerName="test-curl-7763-917d36" Sep 15 21:29:24.646905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.646881 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.649880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.649843 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c3e1d9c1-openshift-default-dockercfg-4xnvv\"" Sep 15 21:29:24.654727 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.654703 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:29:24.673016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.672988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.762217 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762186 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762384 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762607 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flg69\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762607 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762686 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.762783 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.762735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.767826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.767803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.863799 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.863776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.863934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.863808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.863934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.863850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.863934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.863874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flg69\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864292 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864335 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864385 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.864813 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.864785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.865382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.865278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.866726 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.866701 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.866827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.866811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.869718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.869677 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:24.872685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.872666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.872883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.872867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flg69\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69\") pod \"e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.958802 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.958732 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:24.974705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:24.974657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:25.097997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.097963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:25.240248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.240222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:29:25.242056 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:25.242025 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471dcbfd_cfd5_47b6_a6a4_b47b6009a08c.slice/crio-48de356cdf8cb175610c6c03ff345a15ea42ae8b4dd0a928270547e604274635 WatchSource:0}: Error finding container 48de356cdf8cb175610c6c03ff345a15ea42ae8b4dd0a928270547e604274635: Status 404 returned error can't find the container with id 48de356cdf8cb175610c6c03ff345a15ea42ae8b4dd0a928270547e604274635 Sep 15 21:29:25.244367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.244335 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:25.244439 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.244400 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:25.244439 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.244427 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:25.614723 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.614683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerStarted","Data":"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da"} Sep 15 21:29:25.615146 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.614748 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerStarted","Data":"48de356cdf8cb175610c6c03ff345a15ea42ae8b4dd0a928270547e604274635"} Sep 15 21:29:25.615146 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.614862 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7763-917d36" podUID="b59aca09-5ed6-42de-b753-444cac5584ba" containerName="test-curl-7763-917d36" containerID="cri-o://9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd" gracePeriod=2 Sep 15 21:29:25.637161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.637100 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podStartSLOduration=1.637084689 podStartE2EDuration="1.637084689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:25.635850478 +0000 UTC m=+813.860473706" watchObservedRunningTime="2026-09-15 21:29:25.637084689 +0000 UTC m=+813.861707693" Sep 15 21:29:25.709059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.709035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:25.809496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.809465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:25.909714 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.909646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:25.959251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:25.959224 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:26.920023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:26.919998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:26.977191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:26.977027 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:27.008905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.008871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:27.207992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.207914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:27.310736 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.310705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:27.623513 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.623488 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:27.624586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.624571 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:27.812069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.812050 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:27.996383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.996299 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns2wg\" (UniqueName: \"kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg\") pod \"b59aca09-5ed6-42de-b753-444cac5584ba\" (UID: \"b59aca09-5ed6-42de-b753-444cac5584ba\") " Sep 15 21:29:27.998586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:27.998559 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg" (OuterVolumeSpecName: "kube-api-access-ns2wg") pod "b59aca09-5ed6-42de-b753-444cac5584ba" (UID: "b59aca09-5ed6-42de-b753-444cac5584ba"). InnerVolumeSpecName "kube-api-access-ns2wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:29:28.097162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.097120 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2wg\" (UniqueName: \"kubernetes.io/projected/b59aca09-5ed6-42de-b753-444cac5584ba-kube-api-access-ns2wg\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:29:28.216360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.216325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:28.291126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.291043 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59aca09-5ed6-42de-b753-444cac5584ba" path="/var/lib/kubelet/pods/b59aca09-5ed6-42de-b753-444cac5584ba/volumes" Sep 15 21:29:28.628742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.628708 2559 generic.go:356] "Generic (PLEG): container finished" podID="b59aca09-5ed6-42de-b753-444cac5584ba" containerID="9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd" exitCode=137 Sep 15 21:29:28.628895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.628751 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-917d36" Sep 15 21:29:28.628895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.628799 2559 scope.go:117] "RemoveContainer" containerID="9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd" Sep 15 21:29:28.636953 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.636930 2559 scope.go:117] "RemoveContainer" containerID="9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd" Sep 15 21:29:28.637215 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:29:28.637199 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd\": container with ID starting with 9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd not found: ID does not exist" containerID="9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd" Sep 15 21:29:28.637263 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.637222 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd"} err="failed to get container status \"9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd\": rpc error: code = NotFound desc = could not find container \"9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd\": container with ID starting with 9f8c1eea45f77fdffb7700f6ca9b1694ee32351471112d35aa5611e2d3936edd not found: ID does not exist" Sep 15 21:29:28.875042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:28.875010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:29.010931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:29.010842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:29.288261 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:29.288162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:30.262112 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:30.262037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:30.937249 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:30.937209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:31.046494 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:31.046443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:31.280532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:31.279935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:31.363022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:31.362980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:31.556075 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:31.555986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:31.979603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:31.979571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:32.112866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:32.112826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:32.715480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:32.714567 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:32.817242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:32.817184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.110086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.110051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.229677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.229643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.348086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.348057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.449990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.449910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.681049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.681012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.775326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.775216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.853503 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.853441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:33.929711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:33.929682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:34.320047 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:34.320011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:34.503019 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:34.502985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:34.600301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:34.600253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:34.685183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:34.685155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:35.090622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:35.090589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:35.868972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:35.868924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:36.767319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:36.767282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:37.313916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.313875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:37.679568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.679541 2559 generic.go:356] "Generic (PLEG): container finished" podID="677b8361-c00d-4104-86f9-651ce8b839da" containerID="a9bf09c12280b109f84160dcd7c784072aa974e60bb73727d3591fe66c794edd" exitCode=0 Sep 15 21:29:37.679741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.679616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerDied","Data":"a9bf09c12280b109f84160dcd7c784072aa974e60bb73727d3591fe66c794edd"} Sep 15 21:29:37.680059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.680037 2559 scope.go:117] "RemoveContainer" containerID="a9bf09c12280b109f84160dcd7c784072aa974e60bb73727d3591fe66c794edd" Sep 15 21:29:37.681304 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681274 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.681475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerDied","Data":"febdcedd8fef371fb2a2808a99c7fc99c20b7c26352fd67b081970c3c6a41006"} Sep 15 21:29:37.681475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681341 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.681475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681374 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.681475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681286 2559 generic.go:356] "Generic (PLEG): container finished" podID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerID="febdcedd8fef371fb2a2808a99c7fc99c20b7c26352fd67b081970c3c6a41006" exitCode=0 Sep 15 21:29:37.681753 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.681735 2559 scope.go:117] "RemoveContainer" containerID="febdcedd8fef371fb2a2808a99c7fc99c20b7c26352fd67b081970c3c6a41006" Sep 15 21:29:37.682825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.682746 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.682895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.682821 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.682895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.682870 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:37.786681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:37.786647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:38.340207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.340166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:38.690372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.690283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerStarted","Data":"b785abe72ac9876df352a0f8d4d5eee835b7ce4822b30480dee3eaa416382d18"} Sep 15 21:29:38.692655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.692625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerStarted","Data":"0e0dccca693ba20c3dde3eeb19f0e4968d90ccfb1c430e579b4cb0120bc64cb6"} Sep 15 21:29:38.772955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.772911 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:38.773159 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.773051 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:38.773159 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.773097 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:38.973789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.973694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:38.974248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.973826 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:38.974248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:38.973871 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:39.089260 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.089214 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:29:39.243480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.243378 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:29:39.247058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.247041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.250969 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.250948 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bb7867-openshift-default-dockercfg-mxmgl\"" Sep 15 21:29:39.291044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.288739 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:29:39.420144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.420354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.420336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxmm8\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxmm8\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.520892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.521230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.521746 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.521870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.521771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.522802 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.522756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.522959 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.522930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.525100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.525071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.525332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.525310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.536107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.536078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.544339 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.544317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxmm8\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8\") pod \"e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.558263 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.558235 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:39.786354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.785846 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:39.786354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.785910 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:39.912476 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:39.911961 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebeead4_7eb7_4f91_8da6_aeaff08ce88d.slice/crio-25d997e76c76a39539378a166618275cc0d0672d90ff6cbaf8b431269f23b8d3 WatchSource:0}: Error finding container 25d997e76c76a39539378a166618275cc0d0672d90ff6cbaf8b431269f23b8d3: Status 404 returned error can't find the container with id 25d997e76c76a39539378a166618275cc0d0672d90ff6cbaf8b431269f23b8d3 Sep 15 21:29:39.915220 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.915181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:29:39.917079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.917046 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:39.917175 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.917114 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:39.917175 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.917145 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:39.921288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.921266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:39.973956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.973923 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:39.974327 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:39.973979 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:40.024690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.024665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:40.326970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.326933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:40.713283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.713190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerStarted","Data":"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626"} Sep 15 21:29:40.713283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.713258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerStarted","Data":"25d997e76c76a39539378a166618275cc0d0672d90ff6cbaf8b431269f23b8d3"} Sep 15 21:29:40.739242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.739198 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podStartSLOduration=1.7391843059999998 podStartE2EDuration="1.739184306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:40.736892239 +0000 UTC m=+828.961515246" watchObservedRunningTime="2026-09-15 21:29:40.739184306 +0000 UTC m=+828.963807310" Sep 15 21:29:40.772355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.772318 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:40.772643 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.772614 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:40.974080 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.973991 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:40.974655 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:40.974622 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:41.303245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.303157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:41.561340 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.559753 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:41.573485 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.570800 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:41.721696 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.721667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:41.726922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.726898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:29:41.779479 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.778804 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:41.779479 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.778868 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:41.976480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.973550 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:41.976480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:41.973613 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:42.382859 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.382817 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:42.478134 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.478104 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:42.772538 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.772417 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:42.772538 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.772504 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:42.778385 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.778348 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:42.930437 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.930384 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:42.974647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.974189 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:42.974647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:42.974254 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:43.098891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.098818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:43.526039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.525947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:43.773128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.773095 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 15 21:29:43.773401 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.773370 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 15 21:29:43.973697 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.973655 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 15 21:29:43.973901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:43.973721 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 15 21:29:44.412414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.412342 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:44.782926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.782840 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:44.784329 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.784309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:44.785893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.785877 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:29:44.979522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.979493 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:44.981258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.981231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:44.985284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:44.985264 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:29:45.294176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.294144 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:29:45.298383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.298361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.301371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.301350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bb7867-serving-cert\"" Sep 15 21:29:45.302041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.301394 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sgjsd\"" Sep 15 21:29:45.303035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.302596 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 21:29:45.303035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.302777 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 21:29:45.303035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.302919 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 21:29:45.312367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.311985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:29:45.344376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.344345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:45.398776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.398734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.398913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.398833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnls\" (UniqueName: \"kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.398913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.398858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.398913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.398905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.443898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.443872 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:45.500337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.500305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnls\" (UniqueName: \"kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.500551 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.500353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.500551 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.500409 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.500712 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.500617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.501189 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.501165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.502906 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.502879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.503007 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.502926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.518940 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.518912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnls\" (UniqueName: \"kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls\") pod \"maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.566604 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.566526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:45.614207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.614180 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:45.916547 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:45.916520 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:29:45.918959 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:29:45.918930 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3d2192_4f4e_44d3_88a7_ab9a5964687c.slice/crio-b3152dd473366c6033e7bdd938637b1b48abff3a591f7902ac8e9a0756c7b9fb WatchSource:0}: Error finding container b3152dd473366c6033e7bdd938637b1b48abff3a591f7902ac8e9a0756c7b9fb: Status 404 returned error can't find the container with id b3152dd473366c6033e7bdd938637b1b48abff3a591f7902ac8e9a0756c7b9fb Sep 15 21:29:46.023274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:46.023246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:46.119353 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:46.119316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:46.751322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:46.751292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:46.754484 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:46.754437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" event={"ID":"4f3d2192-4f4e-44d3-88a7-ab9a5964687c","Type":"ContainerStarted","Data":"b3152dd473366c6033e7bdd938637b1b48abff3a591f7902ac8e9a0756c7b9fb"} Sep 15 21:29:47.926023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:47.925988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:48.773916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:48.773878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" event={"ID":"4f3d2192-4f4e-44d3-88a7-ab9a5964687c","Type":"ContainerStarted","Data":"59bee60ce37707a0416d71b8edf665e0868c3ef3f14c4fa7f1689d13f6be7cbd"} Sep 15 21:29:48.774082 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:48.774049 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:48.786855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:48.786826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:48.794241 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:48.794194 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" podStartSLOduration=1.120350053 podStartE2EDuration="3.79417939s" totalImagesPullingTime="2.673829337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:48.792741534 +0000 UTC m=+837.017364540" watchObservedRunningTime="2026-09-15 21:29:48.79417939 +0000 UTC m=+837.018802393" Sep 15 21:29:49.545873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:49.545841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:50.444389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.444347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:50.703230 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.703202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:50.792200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.792173 2559 generic.go:356] "Generic (PLEG): container finished" podID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerID="a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da" exitCode=0 Sep 15 21:29:50.792349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.792255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerDied","Data":"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da"} Sep 15 21:29:50.792674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.792661 2559 scope.go:117] "RemoveContainer" containerID="a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da" Sep 15 21:29:50.793821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.793792 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:50.793882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.793863 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:50.793917 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:50.793892 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:29:51.273399 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:51.273365 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:51.800972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:51.800940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerStarted","Data":"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92"} Sep 15 21:29:51.959078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:51.959039 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:51.959252 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:51.959229 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:51.959319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:51.959284 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:52.092496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:52.092447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:52.960021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:52.959984 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:52.960639 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:52.960606 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:53.201307 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:53.201272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:53.960095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:53.960058 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:53.960677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:53.960645 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:54.084084 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:54.084044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:54.791644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:54.791612 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:29:54.959774 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:54.959733 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:54.959962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:54.959797 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:55.011537 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:55.011504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:55.959673 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:55.959629 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:55.959879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:55.959700 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:55.986632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:55.986586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:56.959130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:56.959095 2559 patch_prober.go:28] interesting pod/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Sep 15 21:29:56.959720 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:56.959690 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Sep 15 21:29:57.624162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:57.624122 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:57.963508 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:57.963406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:57.964506 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:57.964484 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:29:58.083242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.083167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.168095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.168060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.243511 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.243398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.329245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.329203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.400862 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.400828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.480772 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.480739 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:29:58.532803 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.532707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.762393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.762358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:58.835741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.835707 2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" secret="" err="secret \"e2e-ait-c3e1d9c1-openshift-default-dockercfg-4xnvv\" not found" Sep 15 21:29:58.932411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:58.932383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:59.030196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.030109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:59.145847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.145762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:59.412852 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.412768 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:29:59.418150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.418127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.420908 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.420888 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5a03be06-gw-openshift-default-dockercfg-qrx68\"" Sep 15 21:29:59.429565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.429546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:29:59.444229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.444198 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:59.533682 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.533638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.534002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.533977 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.534196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.534175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfq7t\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.534412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.534393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.534648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.534629 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.534827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.534810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.535023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.535006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.535204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.535185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.535384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.535367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.636990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.636952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfq7t\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637476 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637476 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.637899 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637876 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.638021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.638021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.637963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.639387 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.639355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.639682 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.639666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.645002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.644979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.645171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.645152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfq7t\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t\") pod \"e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.691020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.690953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:29:59.730369 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.730341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:29:59.841561 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:29:59.841436 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" containerID="cri-o://9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92" gracePeriod=30 Sep 15 21:30:00.041644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.041482 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:30:00.044349 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:30:00.044318 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50fdc2b_bf08_4980_a68d_79dbb9443944.slice/crio-ae20ea37bdf34e3b02f02563ff3427388a5337f71d8c7ad5be818c0c64f363c6 WatchSource:0}: Error finding container ae20ea37bdf34e3b02f02563ff3427388a5337f71d8c7ad5be818c0c64f363c6: Status 404 returned error can't find the container with id ae20ea37bdf34e3b02f02563ff3427388a5337f71d8c7ad5be818c0c64f363c6 Sep 15 21:30:00.046899 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.046862 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:00.046998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.046929 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:00.046998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.046958 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:00.081202 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.081173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:00.196079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.196050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5"] Sep 15 21:30:00.199501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.199485 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" Sep 15 21:30:00.217209 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.217183 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:30:00.219540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.219517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" Sep 15 21:30:00.228437 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.228196 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj"] Sep 15 21:30:00.232823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.232511 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5"] Sep 15 21:30:00.232823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.232627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" Sep 15 21:30:00.245346 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.245327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" Sep 15 21:30:00.267467 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.267420 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj"] Sep 15 21:30:00.510532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.510505 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5"] Sep 15 21:30:00.512265 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:30:00.512235 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e380ae3_ba23_4458_a3a2_1300ee4e1dff.slice/crio-1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751 WatchSource:0}: Error finding container 1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751: Status 404 returned error can't find the container with id 1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751 Sep 15 21:30:00.547848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.547825 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj"] Sep 15 21:30:00.550337 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:30:00.550270 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe65aed_2ad7_401f_b8f0_98fbf518fc89.slice/crio-5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196 WatchSource:0}: Error finding container 5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196: Status 404 returned error can't find the container with id 5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196 Sep 15 21:30:00.769961 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.769924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:00.847979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.847944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" event={"ID":"1e380ae3-ba23-4458-a3a2-1300ee4e1dff","Type":"ContainerStarted","Data":"1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751"} Sep 15 21:30:00.849325 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.849297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" event={"ID":"0fe65aed-2ad7-401f-b8f0-98fbf518fc89","Type":"ContainerStarted","Data":"5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196"} Sep 15 21:30:00.850945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.850916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerStarted","Data":"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7"} Sep 15 21:30:00.851019 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.850954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerStarted","Data":"ae20ea37bdf34e3b02f02563ff3427388a5337f71d8c7ad5be818c0c64f363c6"} Sep 15 21:30:00.881608 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:00.881552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podStartSLOduration=1.881532786 podStartE2EDuration="1.881532786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:00.879144495 +0000 UTC m=+849.103767500" watchObservedRunningTime="2026-09-15 21:30:00.881532786 +0000 UTC m=+849.106155789" Sep 15 21:30:01.695031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.694998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:01.731343 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.731311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:01.737651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.737624 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:01.821370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.821296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:01.862284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.861932 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:01.863097 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.862982 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:01.969562 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:01.969425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:02.071371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:02.071335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:02.485249 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:02.482978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:02.587497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:02.587000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:02.684478 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:02.682960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:03.129363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.129322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:30:03.649425 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.649396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:03.896593 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.896554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" event={"ID":"1e380ae3-ba23-4458-a3a2-1300ee4e1dff","Type":"ContainerStarted","Data":"fda55678b441894f77e60f17fa05fd1aba960f274c89ab3194cfd78402961c23"} Sep 15 21:30:03.898207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.898184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" event={"ID":"0fe65aed-2ad7-401f-b8f0-98fbf518fc89","Type":"ContainerStarted","Data":"f8d8c916faeab5e40955d812988e2c725c8d4421c6b24c822afdf1c39d6d20bd"} Sep 15 21:30:03.914714 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.914673 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" podStartSLOduration=0.880732508 podStartE2EDuration="3.914658512s" totalImagesPullingTime="3.033926004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:03.91302088 +0000 UTC m=+852.137643884" watchObservedRunningTime="2026-09-15 21:30:03.914658512 +0000 UTC m=+852.139281515" Sep 15 21:30:03.929058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:03.929018 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" podStartSLOduration=0.928784492 podStartE2EDuration="3.92900402s" totalImagesPullingTime="3.000219528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:03.927195762 +0000 UTC m=+852.151818769" watchObservedRunningTime="2026-09-15 21:30:03.92900402 +0000 UTC m=+852.153627041" Sep 15 21:30:04.409902 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.409868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:04.554721 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.554671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:04.904347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.904323 2559 generic.go:356] "Generic (PLEG): container finished" podID="1e380ae3-ba23-4458-a3a2-1300ee4e1dff" containerID="fda55678b441894f77e60f17fa05fd1aba960f274c89ab3194cfd78402961c23" exitCode=0 Sep 15 21:30:04.904498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.904402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" event={"ID":"1e380ae3-ba23-4458-a3a2-1300ee4e1dff","Type":"ContainerDied","Data":"fda55678b441894f77e60f17fa05fd1aba960f274c89ab3194cfd78402961c23"} Sep 15 21:30:04.906283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.906264 2559 generic.go:356] "Generic (PLEG): container finished" podID="0fe65aed-2ad7-401f-b8f0-98fbf518fc89" containerID="f8d8c916faeab5e40955d812988e2c725c8d4421c6b24c822afdf1c39d6d20bd" exitCode=0 Sep 15 21:30:04.906379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:04.906354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" event={"ID":"0fe65aed-2ad7-401f-b8f0-98fbf518fc89","Type":"ContainerDied","Data":"f8d8c916faeab5e40955d812988e2c725c8d4421c6b24c822afdf1c39d6d20bd"} Sep 15 21:30:05.004627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.004589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:05.138633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.138593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:05.157398 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.157378 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:30:05.312801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312833 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312864 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312903 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.312993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.312987 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.313147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flg69\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.313147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert\") pod \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\" (UID: \"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c\") " Sep 15 21:30:05.313522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313450 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:30:05.313738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313712 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data" (OuterVolumeSpecName: "istio-data") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:05.313738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:05.313882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313837 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:05.313882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.313851 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:05.315792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.315769 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:30:05.315792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.315781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token" (OuterVolumeSpecName: "istio-token") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:05.315923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.315781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69" (OuterVolumeSpecName: "kube-api-access-flg69") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "kube-api-access-flg69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:05.315923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.315804 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" (UID: "471dcbfd-cfd5-47b6-a6a4-b47b6009a08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414088 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414119 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414129 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414138 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414147 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414157 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414170 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414182 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flg69\" (UniqueName: \"kubernetes.io/projected/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-kube-api-access-flg69\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.414773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.414192 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:05.588791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.588760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:05.913482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.913434 2559 generic.go:356] "Generic (PLEG): container finished" podID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerID="9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92" exitCode=0 Sep 15 21:30:05.913628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.913486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerDied","Data":"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92"} Sep 15 21:30:05.913628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.913512 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" Sep 15 21:30:05.913628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.913554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9" event={"ID":"471dcbfd-cfd5-47b6-a6a4-b47b6009a08c","Type":"ContainerDied","Data":"48de356cdf8cb175610c6c03ff345a15ea42ae8b4dd0a928270547e604274635"} Sep 15 21:30:05.913628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.913577 2559 scope.go:117] "RemoveContainer" containerID="9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92" Sep 15 21:30:05.924270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.924251 2559 scope.go:117] "RemoveContainer" containerID="a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da" Sep 15 21:30:05.934534 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.934510 2559 scope.go:117] "RemoveContainer" containerID="9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92" Sep 15 21:30:05.934937 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:30:05.934909 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92\": container with ID starting with 9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92 not found: ID does not exist" containerID="9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92" Sep 15 21:30:05.935017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.934973 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92"} err="failed to get container status \"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92\": rpc error: code = NotFound desc = could not find container \"9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92\": container with ID starting with 9b3c222652193a5c971c91e1dcd21e9bd27b73bea5cdc2d77067b5654cffed92 not found: ID does not exist" Sep 15 21:30:05.935017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.934997 2559 scope.go:117] "RemoveContainer" containerID="a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da" Sep 15 21:30:05.935545 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:30:05.935370 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da\": container with ID starting with a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da not found: ID does not exist" containerID="a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da" Sep 15 21:30:05.935545 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.935392 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da"} err="failed to get container status \"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da\": rpc error: code = NotFound desc = could not find container \"a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da\": container with ID starting with a9162d3eca17a10d29f8cbd07b999cea19400f985b4f5fef6706222ee82923da not found: ID does not exist" Sep 15 21:30:05.942251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.942225 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:30:05.947582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:05.947559 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c3e1d9c1-openshift-default-7bb5cbf8b-dq9p9"] Sep 15 21:30:06.124028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.124005 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" Sep 15 21:30:06.126063 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.126045 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" Sep 15 21:30:06.293618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.293537 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" path="/var/lib/kubelet/pods/471dcbfd-cfd5-47b6-a6a4-b47b6009a08c/volumes" Sep 15 21:30:06.561821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.561728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:06.921488 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.921443 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" Sep 15 21:30:06.921488 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.921481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5" event={"ID":"1e380ae3-ba23-4458-a3a2-1300ee4e1dff","Type":"ContainerDied","Data":"1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751"} Sep 15 21:30:06.921801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.921539 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1df00cf99e6f5d66e73d76924540ede93056025ca5a8e4af304fe4cc5e3af751" Sep 15 21:30:06.922868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.922843 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" Sep 15 21:30:06.922983 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.922845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-c94bj" event={"ID":"0fe65aed-2ad7-401f-b8f0-98fbf518fc89","Type":"ContainerDied","Data":"5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196"} Sep 15 21:30:06.922983 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:06.922957 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef7ba0582c51fc32dcfb7cfe23f1db691564f35e134175c935b526efcf7b196" Sep 15 21:30:07.663166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:07.663132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:08.470334 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:08.470297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:08.883883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:08.883833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:08.987892 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:08.987851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:09.594587 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:09.594556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:10.504937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.504901 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:10.945999 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.945970 2559 generic.go:356] "Generic (PLEG): container finished" podID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerID="318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626" exitCode=0 Sep 15 21:30:10.946161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.946043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerDied","Data":"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626"} Sep 15 21:30:10.946444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.946429 2559 scope.go:117] "RemoveContainer" containerID="318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626" Sep 15 21:30:10.947688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.947661 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:10.947752 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.947730 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:10.947795 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:10.947760 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:11.721555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:11.721518 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:30:11.952841 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:11.952810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerStarted","Data":"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e"} Sep 15 21:30:12.559637 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:12.559604 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:30:12.560759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:12.560730 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 21:30:12.560966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:12.560940 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 21:30:13.559320 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:13.559283 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 21:30:13.559918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:13.559886 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 21:30:14.559514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:14.559479 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 21:30:14.560077 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:14.560045 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 21:30:14.993920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:14.993884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:15.094251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:15.094218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:15.559137 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:15.559095 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 21:30:15.559379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:15.559163 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 21:30:16.559347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:16.559308 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 15 21:30:16.559824 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:16.559388 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 15 21:30:17.585860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:17.585816 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:30:17.586649 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:17.586621 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:30:17.587428 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:17.587409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:30:23.641264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:23.641230 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:30:25.502909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:25.502876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:26.036244 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.036213 2559 generic.go:356] "Generic (PLEG): container finished" podID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerID="96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7" exitCode=0 Sep 15 21:30:26.036413 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.036290 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerDied","Data":"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7"} Sep 15 21:30:26.036734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.036720 2559 scope.go:117] "RemoveContainer" containerID="96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7" Sep 15 21:30:26.037971 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.037946 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:26.038038 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.038011 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:26.038075 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.038043 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:30:26.997328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:26.997283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:27.043353 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:27.043321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerStarted","Data":"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428"} Sep 15 21:30:27.731639 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:27.731596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:27.732822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:27.732797 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:27.732934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:27.732881 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:27.823535 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:27.823501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:28.693252 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:28.693218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:28.731359 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:28.731328 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:28.731554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:28.731385 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:29.469022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:29.468950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:29.731711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:29.731622 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:29.731711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:29.731698 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:30.374375 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:30.374339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:30.730967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:30.730884 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:30.730967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:30.730949 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:31.212276 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:31.212238 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:31.731127 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:31.731096 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:31.731297 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:31.731151 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:31.862423 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:31.862384 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:32.066326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:32.066172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:32.731291 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:32.731253 2559 patch_prober.go:28] interesting pod/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 15 21:30:32.731778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:32.731312 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 15 21:30:32.810146 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:32.810108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:33.569014 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:33.568984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:33.735347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:33.735319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:33.736251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:33.736233 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:34.230281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.230251 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:34.299168 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.299139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:34.399482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.399425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:34.465342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.465311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:34.574268 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.574195 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:30:34.587427 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.587395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:34.903821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:34.903789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:35.025850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:35.025818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:35.097048 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:35.096993 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" containerID="cri-o://686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428" gracePeriod=30 Sep 15 21:30:35.365805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:35.365772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:35.762045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:35.761951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:36.381506 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:36.381419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:36.732017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:36.731914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:37.454686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:37.454635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:37.562434 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:37.562396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:37.658986 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:37.658932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:38.548988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.548955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:38.703572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703534 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:30:38.703907 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703894 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703908 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703916 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe65aed-2ad7-401f-b8f0-98fbf518fc89" containerName="cleanup" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703922 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe65aed-2ad7-401f-b8f0-98fbf518fc89" containerName="cleanup" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703937 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e380ae3-ba23-4458-a3a2-1300ee4e1dff" containerName="cleanup" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703942 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e380ae3-ba23-4458-a3a2-1300ee4e1dff" containerName="cleanup" Sep 15 21:30:38.703952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703952 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.704129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.703957 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.704129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.704028 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.704129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.704039 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="471dcbfd-cfd5-47b6-a6a4-b47b6009a08c" containerName="istio-proxy" Sep 15 21:30:38.704129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.704045 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe65aed-2ad7-401f-b8f0-98fbf518fc89" containerName="cleanup" Sep 15 21:30:38.704129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.704051 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e380ae3-ba23-4458-a3a2-1300ee4e1dff" containerName="cleanup" Sep 15 21:30:38.707414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.707395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.711326 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.711304 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"kube-root-ca.crt\"" Sep 15 21:30:38.712644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.712619 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"openshift-service-ca.crt\"" Sep 15 21:30:38.712801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.712646 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"default-dockercfg-g8wsl\"" Sep 15 21:30:38.712801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.712743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"simulator-w0-b99d7b-kserve-self-signed-certs\"" Sep 15 21:30:38.720430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.720411 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:30:38.754337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.754436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcc58\" (UniqueName: \"kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.754436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.754436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754399 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.754436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.754650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.754487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.830498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.830450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:38.855846 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.855821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.855956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.855853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcc58\" (UniqueName: \"kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.855956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.855880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.855956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.855902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.856140 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.856119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.856273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.856242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.856273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.856262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.856391 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.856345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.856595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.856575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.858177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.858154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.858771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.858751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:38.865306 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:38.865279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcc58\" (UniqueName: \"kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58\") pod \"simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:39.016835 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:39.016798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:39.298050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:39.298023 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:30:39.299492 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:30:39.299451 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e29fee9_f4c5_49b9_a2ed_9a2c398a5246.slice/crio-c6b52d953f39ac23bbd5bb55e016e5d61ab2108a9e0a5aceb2fa1ff5185cb6b2 WatchSource:0}: Error finding container c6b52d953f39ac23bbd5bb55e016e5d61ab2108a9e0a5aceb2fa1ff5185cb6b2: Status 404 returned error can't find the container with id c6b52d953f39ac23bbd5bb55e016e5d61ab2108a9e0a5aceb2fa1ff5185cb6b2 Sep 15 21:30:40.034507 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.034474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:30:40.124031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.123958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" event={"ID":"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246","Type":"ContainerStarted","Data":"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c"} Sep 15 21:30:40.124031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.124000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" event={"ID":"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246","Type":"ContainerStarted","Data":"c6b52d953f39ac23bbd5bb55e016e5d61ab2108a9e0a5aceb2fa1ff5185cb6b2"} Sep 15 21:30:40.150516 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.150474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" podStartSLOduration=2.150441582 podStartE2EDuration="2.150441582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:40.149304267 +0000 UTC m=+888.373927270" watchObservedRunningTime="2026-09-15 21:30:40.150441582 +0000 UTC m=+888.375064586" Sep 15 21:30:40.361403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.361360 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:40.470064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.469978 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470073 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfq7t\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470120 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470175 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470334 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy\") pod \"f50fdc2b-bf08-4980-a68d-79dbb9443944\" (UID: \"f50fdc2b-bf08-4980-a68d-79dbb9443944\") " Sep 15 21:30:40.470610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:40.470823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:40.470905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470791 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:40.470905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470862 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:30:40.471013 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.470988 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data" (OuterVolumeSpecName: "istio-data") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:40.471076 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.471005 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:40.472473 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.472421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:30:40.472617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.472601 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token" (OuterVolumeSpecName: "istio-token") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:40.472876 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.472860 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t" (OuterVolumeSpecName: "kube-api-access-vfq7t") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "kube-api-access-vfq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:30:40.472972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.472951 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f50fdc2b-bf08-4980-a68d-79dbb9443944" (UID: "f50fdc2b-bf08-4980-a68d-79dbb9443944"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:30:40.572025 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.571998 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572025 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572024 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572034 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572046 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572058 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572066 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572074 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572081 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50fdc2b-bf08-4980-a68d-79dbb9443944-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.572222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.572089 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfq7t\" (UniqueName: \"kubernetes.io/projected/f50fdc2b-bf08-4980-a68d-79dbb9443944-kube-api-access-vfq7t\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:30:40.595394 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:40.595366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:41.131638 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.131607 2559 generic.go:356] "Generic (PLEG): container finished" podID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerID="686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428" exitCode=0 Sep 15 21:30:41.132000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.131656 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" Sep 15 21:30:41.132000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.131686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerDied","Data":"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428"} Sep 15 21:30:41.132000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.131733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb" event={"ID":"f50fdc2b-bf08-4980-a68d-79dbb9443944","Type":"ContainerDied","Data":"ae20ea37bdf34e3b02f02563ff3427388a5337f71d8c7ad5be818c0c64f363c6"} Sep 15 21:30:41.132000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.131804 2559 scope.go:117] "RemoveContainer" containerID="686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428" Sep 15 21:30:41.142789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.142565 2559 scope.go:117] "RemoveContainer" containerID="96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7" Sep 15 21:30:41.151901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.151883 2559 scope.go:117] "RemoveContainer" containerID="686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428" Sep 15 21:30:41.152166 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:30:41.152147 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428\": container with ID starting with 686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428 not found: ID does not exist" containerID="686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428" Sep 15 21:30:41.152245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.152176 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428"} err="failed to get container status \"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428\": rpc error: code = NotFound desc = could not find container \"686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428\": container with ID starting with 686dd80f894a5392674f430c48515a3a2af2077f71277383820e00d3fcd83428 not found: ID does not exist" Sep 15 21:30:41.152245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.152201 2559 scope.go:117] "RemoveContainer" containerID="96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7" Sep 15 21:30:41.152442 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:30:41.152425 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7\": container with ID starting with 96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7 not found: ID does not exist" containerID="96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7" Sep 15 21:30:41.152524 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.152450 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7"} err="failed to get container status \"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7\": rpc error: code = NotFound desc = could not find container \"96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7\": container with ID starting with 96748dd2c6bd056a32a594b21269e178d80c1b44f7ab1dfdeea26a447eeb45a7 not found: ID does not exist" Sep 15 21:30:41.177680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.177657 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:30:41.189702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:41.189676 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5a03be06-gw-openshift-default-7f6ff89f8d-bf6sb"] Sep 15 21:30:42.200331 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:42.200275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:42.291279 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:42.291250 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" path="/var/lib/kubelet/pods/f50fdc2b-bf08-4980-a68d-79dbb9443944/volumes" Sep 15 21:30:43.407509 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:43.407477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:43.484240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:43.484172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:44.904556 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:44.904440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:30:49.017145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:49.017109 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:49.017145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:49.017148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:49.030017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:49.029992 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:49.186543 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:49.186516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:30:53.321772 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:30:53.321739 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:31:02.590887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:02.590848 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:03.569820 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:03.569788 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:04.356044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:04.356006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:05.169845 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:05.169785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:06.030343 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:06.030309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:06.734511 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:06.734477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:07.581663 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:07.581629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:08.333056 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:08.333021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:09.193171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.193129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:09.601952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.601920 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:31:09.602329 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602315 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.602406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602329 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.602406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602338 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.602406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602344 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.602406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602405 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.602591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.602416 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50fdc2b-bf08-4980-a68d-79dbb9443944" containerName="istio-proxy" Sep 15 21:31:09.606694 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.606671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.610766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.610731 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"openshift-service-ca.crt\"" Sep 15 21:31:09.610766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.610745 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"kube-root-ca.crt\"" Sep 15 21:31:09.610947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.610771 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"simulator-w4-ac4411-kserve-self-signed-certs\"" Sep 15 21:31:09.610947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.610798 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"default-dockercfg-brd6c\"" Sep 15 21:31:09.617762 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.617740 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:31:09.630306 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.630445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.630445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.630445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.630610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.630610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.630514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgcv\" (UniqueName: \"kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.640805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.640781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:09.731805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.731763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.731805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.731807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgcv\" (UniqueName: \"kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732087 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.731902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732087 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.731934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732087 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.731972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.732105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732361 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.732329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732361 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.732349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.732551 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.732506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.734748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.734724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.734987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.734969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.739993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.739973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgcv\" (UniqueName: \"kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv\") pod \"simulator-w4-ac4411-kserve-8586c556c5-spjjk\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.848626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.848594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:09.917695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.917560 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:09.931817 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.931753 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:09.990140 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:09.990048 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:10.071390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.071354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:10.201845 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.201819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:31:10.203832 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:31:10.203804 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa936ca7_57ed_4f0b_9d38_de5d65e99fe5.slice/crio-316bc5c89280c9f598cad4ebdf38d85d6e6a42b86f4040b40dfd28a7931d2b29 WatchSource:0}: Error finding container 316bc5c89280c9f598cad4ebdf38d85d6e6a42b86f4040b40dfd28a7931d2b29: Status 404 returned error can't find the container with id 316bc5c89280c9f598cad4ebdf38d85d6e6a42b86f4040b40dfd28a7931d2b29 Sep 15 21:31:10.290383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.290355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" event={"ID":"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5","Type":"ContainerStarted","Data":"99733f38d469198f0ede59ffbd18a7134b5317a99ca064ab9940c44b2b703f47"} Sep 15 21:31:10.290518 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.290402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" event={"ID":"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5","Type":"ContainerStarted","Data":"316bc5c89280c9f598cad4ebdf38d85d6e6a42b86f4040b40dfd28a7931d2b29"} Sep 15 21:31:10.307662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.307604 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" podStartSLOduration=1.307586568 podStartE2EDuration="1.307586568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:10.305828001 +0000 UTC m=+918.530451006" watchObservedRunningTime="2026-09-15 21:31:10.307586568 +0000 UTC m=+918.532209576" Sep 15 21:31:10.325346 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.325320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:10.532909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:10.532830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:11.200688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:11.200651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:11.357809 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:11.357760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:11.640148 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:11.640115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:12.033192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:12.033117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:12.177078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:12.177047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:12.320308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:12.320235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:13.199739 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:13.199700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:13.968541 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:13.968502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:14.791811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:14.791781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:14.995150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:14.995116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:15.081303 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:15.081273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:15.784139 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:15.784111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:16.531219 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:16.531182 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:31:16.705787 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:16.705756 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:17.565215 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:17.565185 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:18.294546 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.294511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:18.514849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.514812 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:31:18.518625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.518608 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.521248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.521220 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-10c104\"/\"kube-root-ca.crt\"" Sep 15 21:31:18.521374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.521254 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-10c104\"/\"openshift-service-ca.crt\"" Sep 15 21:31:18.521374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.521341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-10c104\"/\"default-dockercfg-qsc7f\"" Sep 15 21:31:18.522354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.522335 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-10c104\"/\"autores-model-10c104-kserve-self-signed-certs\"" Sep 15 21:31:18.528536 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.528515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:31:18.631743 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.632177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.632177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631873 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxtfx\" (UniqueName: \"kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.632177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.632177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.632177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.631967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtfx\" (UniqueName: \"kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733817 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733863 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.733863 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.733845 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.735766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.735744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.735947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.735929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.742016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.741995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtfx\" (UniqueName: \"kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx\") pod \"autores-model-10c104-kserve-6c45c5879-r4trs\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.829351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.829327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:18.924926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:18.924877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:19.053537 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.053496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:19.106923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.106897 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:31:19.108531 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:31:19.108499 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6afc145f_8d3b_4649_94d2_9eb8b2fb5bcb.slice/crio-95e52fae116f9797f6ae15bf058dc04ec7b92fa6d8e2b25141ce43ebcd003929 WatchSource:0}: Error finding container 95e52fae116f9797f6ae15bf058dc04ec7b92fa6d8e2b25141ce43ebcd003929: Status 404 returned error can't find the container with id 95e52fae116f9797f6ae15bf058dc04ec7b92fa6d8e2b25141ce43ebcd003929 Sep 15 21:31:19.336672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.336640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" event={"ID":"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb","Type":"ContainerStarted","Data":"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a"} Sep 15 21:31:19.336848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.336682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" event={"ID":"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb","Type":"ContainerStarted","Data":"95e52fae116f9797f6ae15bf058dc04ec7b92fa6d8e2b25141ce43ebcd003929"} Sep 15 21:31:19.358496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.358416 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" podStartSLOduration=1.358395383 podStartE2EDuration="1.358395383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:19.357827883 +0000 UTC m=+927.582450888" watchObservedRunningTime="2026-09-15 21:31:19.358395383 +0000 UTC m=+927.583018396" Sep 15 21:31:19.918337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.918302 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:19.918730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.918350 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:19.930792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:19.930768 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:20.176626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.176541 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:31:20.180140 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.180116 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.183540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.183519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d578a96c-serving-cert\"" Sep 15 21:31:20.195677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.195651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:31:20.243424 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.243392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:20.248702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.248676 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.248805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.248755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4vx\" (UniqueName: \"kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.248853 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.248822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.248900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.248882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.343580 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.343533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:20.349338 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.349313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.349501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.349379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.349501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.349413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.349501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.349480 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4vx\" (UniqueName: \"kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.349672 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:31:20.349611 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-serving-cert: secret "maas-api-e2e-derive-d578a96c-serving-cert" not found Sep 15 21:31:20.349720 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:31:20.349694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls podName:7a0acbe3-9752-4dca-b439-d58f09d23fbe nodeName:}" failed. No retries permitted until 2026-09-15 21:31:20.849674949 +0000 UTC m=+929.074297931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls") pod "maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" (UID: "7a0acbe3-9752-4dca-b439-d58f09d23fbe") : secret "maas-api-e2e-derive-d578a96c-serving-cert" not found Sep 15 21:31:20.350035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.350007 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.352450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.352415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.355962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.355944 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:31:20.362172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.362146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4vx\" (UniqueName: \"kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.464393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.464325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:20.853919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.853884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.856106 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.856081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") pod \"maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:20.875740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.875716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:20.995118 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:20.995085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:21.091640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:21.091604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:21.369201 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:21.369174 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:31:21.370508 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:31:21.370478 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0acbe3_9752_4dca_b439_d58f09d23fbe.slice/crio-a06c372075b3d2169428d985504056731c69111e88bce3556d5b4814f13d9a25 WatchSource:0}: Error finding container a06c372075b3d2169428d985504056731c69111e88bce3556d5b4814f13d9a25: Status 404 returned error can't find the container with id a06c372075b3d2169428d985504056731c69111e88bce3556d5b4814f13d9a25 Sep 15 21:31:22.361722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:22.361669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" event={"ID":"7a0acbe3-9752-4dca-b439-d58f09d23fbe","Type":"ContainerStarted","Data":"c41ff512411c16be72f5707412f3f5f9be7e198b796ee328d4f833adc3991d82"} Sep 15 21:31:22.362296 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:22.361772 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:22.362296 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:22.361795 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" event={"ID":"7a0acbe3-9752-4dca-b439-d58f09d23fbe","Type":"ContainerStarted","Data":"a06c372075b3d2169428d985504056731c69111e88bce3556d5b4814f13d9a25"} Sep 15 21:31:22.380801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:22.380760 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" podStartSLOduration=2.047155496 podStartE2EDuration="2.380744888s" totalImagesPullingTime="333.589392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:22.379638549 +0000 UTC m=+930.604261554" watchObservedRunningTime="2026-09-15 21:31:22.380744888 +0000 UTC m=+930.605367885" Sep 15 21:31:28.374711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:28.374679 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:28.830286 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:28.830254 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:28.830493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:28.830342 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:28.842720 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:28.842700 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:29.408331 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:29.408305 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:31:36.622263 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:36.622227 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:31:42.860695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:42.860661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:44.404237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:44.404201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:44.508665 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:44.508629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:45.421895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:45.421865 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:46.247218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:46.247187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:47.273552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:47.273519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:48.044864 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:48.044835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:49.000692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:49.000657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:49.885397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:49.885362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:50.645167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:50.645137 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:52.162669 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:52.162633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:55.073292 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:55.073259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:55.879838 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:55.879803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:56.866686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:56.866646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:58.241193 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:58.241163 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:31:59.385136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.385101 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:31:59.385625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.385364 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" podUID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" containerName="maas-api" containerID="cri-o://c41ff512411c16be72f5707412f3f5f9be7e198b796ee328d4f833adc3991d82" gracePeriod=30 Sep 15 21:31:59.554373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.554341 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" containerID="c41ff512411c16be72f5707412f3f5f9be7e198b796ee328d4f833adc3991d82" exitCode=0 Sep 15 21:31:59.554555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.554419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" event={"ID":"7a0acbe3-9752-4dca-b439-d58f09d23fbe","Type":"ContainerDied","Data":"c41ff512411c16be72f5707412f3f5f9be7e198b796ee328d4f833adc3991d82"} Sep 15 21:31:59.612050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.612011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:59.658023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.657999 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:31:59.694365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694334 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca\") pod \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " Sep 15 21:31:59.694519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694495 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs\") pod \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " Sep 15 21:31:59.694603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694543 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk4vx\" (UniqueName: \"kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx\") pod \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " Sep 15 21:31:59.694603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694576 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") pod \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\" (UID: \"7a0acbe3-9752-4dca-b439-d58f09d23fbe\") " Sep 15 21:31:59.694710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a0acbe3-9752-4dca-b439-d58f09d23fbe" (UID: "7a0acbe3-9752-4dca-b439-d58f09d23fbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:31:59.694970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.694931 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0acbe3-9752-4dca-b439-d58f09d23fbe-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.696984 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.696946 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a0acbe3-9752-4dca-b439-d58f09d23fbe" (UID: "7a0acbe3-9752-4dca-b439-d58f09d23fbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:31:59.696984 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.696971 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a0acbe3-9752-4dca-b439-d58f09d23fbe" (UID: "7a0acbe3-9752-4dca-b439-d58f09d23fbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:31:59.697539 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.697505 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx" (OuterVolumeSpecName: "kube-api-access-jk4vx") pod "7a0acbe3-9752-4dca-b439-d58f09d23fbe" (UID: "7a0acbe3-9752-4dca-b439-d58f09d23fbe"). InnerVolumeSpecName "kube-api-access-jk4vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:31:59.712263 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.712232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:31:59.796028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.795997 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.796028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.796019 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0acbe3-9752-4dca-b439-d58f09d23fbe-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.796028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.796028 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4vx\" (UniqueName: \"kubernetes.io/projected/7a0acbe3-9752-4dca-b439-d58f09d23fbe-kube-api-access-jk4vx\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:31:59.805218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:31:59.805193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:00.464222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.464181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:00.561390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.561313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" event={"ID":"7a0acbe3-9752-4dca-b439-d58f09d23fbe","Type":"ContainerDied","Data":"a06c372075b3d2169428d985504056731c69111e88bce3556d5b4814f13d9a25"} Sep 15 21:32:00.561390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.561386 2559 scope.go:117] "RemoveContainer" containerID="c41ff512411c16be72f5707412f3f5f9be7e198b796ee328d4f833adc3991d82" Sep 15 21:32:00.561625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.561323 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb" Sep 15 21:32:00.590941 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.590914 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:32:00.598777 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:00.598754 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d578a96c-84fb9c9894-7h7kb"] Sep 15 21:32:02.290495 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:02.290442 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" path="/var/lib/kubelet/pods/7a0acbe3-9752-4dca-b439-d58f09d23fbe/volumes" Sep 15 21:32:02.390192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:02.390156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:02.492063 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:02.492016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:03.200916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:03.200888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:04.070401 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:04.070337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:04.794686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:04.794645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:05.562568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:05.562533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:06.400656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:06.400624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:06.527888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:06.527859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:06.628315 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:06.628283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:06.689204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:06.689119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:07.019966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:07.019879 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:07.688656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:07.688617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:08.432251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:08.432217 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:08.552654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:08.552623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:08.652943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:08.652912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:08.745993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:08.745922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:09.106305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:09.106271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:09.474544 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:09.474443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:10.368517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:10.368477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:12.004659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:12.004623 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:32:12.051373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:12.051345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:18.329882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:18.329793 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:32:27.023315 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.023257 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" podUID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" containerName="main" containerID="cri-o://0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a" gracePeriod=45 Sep 15 21:32:27.270861 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.270833 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:32:27.359313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359283 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxtfx\" (UniqueName: \"kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359359 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359384 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359443 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359567 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359599 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs\") pod \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\" (UID: \"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb\") " Sep 15 21:32:27.359659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:32:27.359823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359659 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home" (OuterVolumeSpecName: "home") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:32:27.359823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359766 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:32:27.359975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359951 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.360024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359978 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.360024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.359991 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.361755 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.361731 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:32:27.361862 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.361729 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx" (OuterVolumeSpecName: "kube-api-access-nxtfx") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "kube-api-access-nxtfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:32:27.361904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.361864 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm" (OuterVolumeSpecName: "dshm") pod "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" (UID: "6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:32:27.461372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.461349 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.461372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.461369 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.461530 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.461380 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxtfx\" (UniqueName: \"kubernetes.io/projected/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb-kube-api-access-nxtfx\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:32:27.708640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.708558 2559 generic.go:356] "Generic (PLEG): container finished" podID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" containerID="0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a" exitCode=0 Sep 15 21:32:27.708640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.708607 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" Sep 15 21:32:27.708640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.708626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" event={"ID":"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb","Type":"ContainerDied","Data":"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a"} Sep 15 21:32:27.708832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.708665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs" event={"ID":"6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb","Type":"ContainerDied","Data":"95e52fae116f9797f6ae15bf058dc04ec7b92fa6d8e2b25141ce43ebcd003929"} Sep 15 21:32:27.708832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.708688 2559 scope.go:117] "RemoveContainer" containerID="0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a" Sep 15 21:32:27.717839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.717817 2559 scope.go:117] "RemoveContainer" containerID="0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a" Sep 15 21:32:27.718107 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:32:27.718086 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a\": container with ID starting with 0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a not found: ID does not exist" containerID="0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a" Sep 15 21:32:27.718156 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.718114 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a"} err="failed to get container status \"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a\": rpc error: code = NotFound desc = could not find container \"0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a\": container with ID starting with 0b0c1b3afa48d7343f3b9069ff680787bffbeab0e8e16b7df739598162ede58a not found: ID does not exist" Sep 15 21:32:27.731422 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.731399 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:32:27.736792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:27.736762 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-10c104/autores-model-10c104-kserve-6c45c5879-r4trs"] Sep 15 21:32:28.292057 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:28.292027 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" path="/var/lib/kubelet/pods/6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb/volumes" Sep 15 21:32:32.960031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:32.959983 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:34.998675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:34.998635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:40.916160 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:40.916122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:40.984471 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:40.982317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:41.105369 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:41.105332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:41.209358 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:41.209277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:41.746236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:41.746203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:47.424055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:47.424017 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:32:49.163881 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:49.163847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:49.411594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:49.411562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:49.823593 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:49.823554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:50.124872 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:50.124828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:50.512252 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:50.512173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:52.179884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:52.179845 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:52.965665 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:52.965627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:53.047966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:53.047933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:53.274045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:53.273966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:55.031126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:55.031095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:55.131593 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:55.131562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:55.874267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:55.874234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:56.608344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:56.608311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:56.712200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:56.712166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:56.960957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:56.960877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:57.081510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:57.081478 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:57.184155 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:57.184120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:32:57.294585 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:32:57.294487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:02.731092 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:02.731058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:02.857942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:02.857909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:02.966355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:02.966299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:03.056318 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:03.056224 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:04.721162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721133 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:33:04.721560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721501 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" containerName="main" Sep 15 21:33:04.721560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721514 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" containerName="main" Sep 15 21:33:04.721560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721533 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" containerName="maas-api" Sep 15 21:33:04.721560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721538 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" containerName="maas-api" Sep 15 21:33:04.721713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721599 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0acbe3-9752-4dca-b439-d58f09d23fbe" containerName="maas-api" Sep 15 21:33:04.721713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.721610 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afc145f-8d3b-4649-94d2-9eb8b2fb5bcb" containerName="main" Sep 15 21:33:04.725947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.725921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.728976 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.728953 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"premium-w4-ac4411-kserve-self-signed-certs\"" Sep 15 21:33:04.736115 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.736093 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:33:04.790366 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc6j\" (UniqueName: \"kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.790554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.790554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.790554 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790531 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.790718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790567 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.790766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.790714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.816123 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.816099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:04.891172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc6j\" (UniqueName: \"kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891544 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.891901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.891822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.893426 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.893403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.893897 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.893874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:04.904051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:04.904027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc6j\" (UniqueName: \"kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j\") pod \"premium-w4-ac4411-kserve-577bdd8fb6-xj4xf\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:05.036522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:05.036416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:05.326228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:05.326197 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:33:05.329399 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:33:05.329377 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod950d6cb1_65c7_4a68_854e_79eb133a6e25.slice/crio-ddd9dfda9507dc096fe984dd33730ac8cd333dcd67fea8124b51fe7ca01222b1 WatchSource:0}: Error finding container ddd9dfda9507dc096fe984dd33730ac8cd333dcd67fea8124b51fe7ca01222b1: Status 404 returned error can't find the container with id ddd9dfda9507dc096fe984dd33730ac8cd333dcd67fea8124b51fe7ca01222b1 Sep 15 21:33:05.904806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:05.904769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" event={"ID":"950d6cb1-65c7-4a68-854e-79eb133a6e25","Type":"ContainerStarted","Data":"8d92ff5fd39ed4e9638377041ff4b73c8fb2d2c3ca9e05469644dfd64d4a29b5"} Sep 15 21:33:05.905149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:05.904816 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" event={"ID":"950d6cb1-65c7-4a68-854e-79eb133a6e25","Type":"ContainerStarted","Data":"ddd9dfda9507dc096fe984dd33730ac8cd333dcd67fea8124b51fe7ca01222b1"} Sep 15 21:33:05.923082 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:05.923034 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" podStartSLOduration=1.923017276 podStartE2EDuration="1.923017276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:05.92149645 +0000 UTC m=+1034.146119452" watchObservedRunningTime="2026-09-15 21:33:05.923017276 +0000 UTC m=+1034.147640279" Sep 15 21:33:10.027696 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:10.027660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:33:13.529086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:13.529051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:14.303364 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:14.303319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:15.037525 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:15.037483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:15.037525 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:15.037533 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:15.049786 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:15.049761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:15.965767 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:15.965732 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:33:18.827351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:18.827317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:19.514945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:19.514905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:19.778528 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:19.778412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:20.289780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:20.289749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:20.483794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:20.483766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:20.589359 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:20.589325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:20.680782 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:20.680751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:20.782662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:20.782632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.028472 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.028371 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:33:21.032052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.032029 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.034838 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.034816 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bb7867\"/\"test-model-bb7867-kserve-self-signed-certs\"" Sep 15 21:33:21.036046 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.036022 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bb7867\"/\"openshift-service-ca.crt\"" Sep 15 21:33:21.036133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.036053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bb7867\"/\"default-dockercfg-25t4g\"" Sep 15 21:33:21.036133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.036075 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bb7867\"/\"kube-root-ca.crt\"" Sep 15 21:33:21.042739 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.042717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:33:21.060310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.060271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.150891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.150862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.151061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.150897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.151061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.150933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.151061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.151028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnq42\" (UniqueName: \"kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.151168 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.151072 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.151168 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.151096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.241636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.241593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.251960 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.251931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.251974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252239 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnq42\" (UniqueName: \"kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252239 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252239 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.252578 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.252557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.254635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.254613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.255017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.255001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.261167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.261104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnq42\" (UniqueName: \"kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42\") pod \"test-model-bb7867-kserve-7c748c49c-bx6lr\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.342681 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.342659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:21.416058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.416016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.504267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.504235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.601711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.601619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:21.639270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.639246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:33:21.640795 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:33:21.640768 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae008835_baac_403f_9eb4_4784935f0b8d.slice/crio-a450e6485233b4c9b0131496984e76764184319284269bb1b8473b298e150f09 WatchSource:0}: Error finding container a450e6485233b4c9b0131496984e76764184319284269bb1b8473b298e150f09: Status 404 returned error can't find the container with id a450e6485233b4c9b0131496984e76764184319284269bb1b8473b298e150f09 Sep 15 21:33:21.986754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.986675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" event={"ID":"ae008835-baac-403f-9eb4-4784935f0b8d","Type":"ContainerStarted","Data":"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a"} Sep 15 21:33:21.986754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:21.986715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" event={"ID":"ae008835-baac-403f-9eb4-4784935f0b8d","Type":"ContainerStarted","Data":"a450e6485233b4c9b0131496984e76764184319284269bb1b8473b298e150f09"} Sep 15 21:33:22.005540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:22.005492 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" podStartSLOduration=1.005476093 podStartE2EDuration="1.005476093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:22.005082598 +0000 UTC m=+1050.229705602" watchObservedRunningTime="2026-09-15 21:33:22.005476093 +0000 UTC m=+1050.230099091" Sep 15 21:33:22.395904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:22.395861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:22.997686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:22.997646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:23.163674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:23.163636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:23.388010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:23.387980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:28.945373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:28.945337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:29.006197 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:29.006166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:29.645130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:29.645094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:31.343796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:31.343742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:31.343796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:31.343810 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:31.356801 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:31.356777 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:32.048814 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:32.048788 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:33:32.229382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:32.229343 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:33:35.457244 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:35.457211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:35.525975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:35.525943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:35.682158 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:35.682129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:35.987045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:35.987007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:36.081636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:36.081603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:41.786364 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:41.786331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:42.015552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:42.015521 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:42.118971 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:42.118943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:42.787640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:42.787593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:43.514611 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:43.514583 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:44.245185 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:44.245147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:45.132731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:45.132652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:45.864035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:45.863984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:45.963911 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:45.963880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:46.144614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:46.144533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:46.426891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:46.426808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:46.518134 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:46.518100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:33:57.649514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:33:57.649479 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:34:09.373152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:09.373116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:10.386406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:10.386369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:11.125748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:11.125709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:12.037926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:12.037888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:12.139036 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:12.139000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:12.660369 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:12.660328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:12.759589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:12.759554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:12.864298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:12.864266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:13.100698 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:13.100660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:13.673888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:13.673854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:14.573760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:14.573724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:19.286337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:19.286289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:19.352883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:19.352855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:20.574580 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:20.574549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:21.340716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:21.340681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:22.095873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:22.095829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:23.033260 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:23.033225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:23.757512 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:23.757441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:24.766767 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:24.766716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:25.484926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:25.484891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:25.837803 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:25.837771 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:34:30.219109 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.219074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.287721 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.287668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.353517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.353486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.448708 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.448682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.532677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.532553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.801754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.801672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:30.987261 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:30.987224 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:31.887069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:31.887040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:32.687193 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:32.687157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:33.588783 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:33.588748 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:34.576575 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:34.576537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:35.428759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:35.428726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:36.865537 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:36.865504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:36.965150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:36.965117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:37.128371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:37.128287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:37.445392 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:37.445304 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:37.535312 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:37.535285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:37.638839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:37.638806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:40.349762 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:40.349725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:43.486438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:43.486405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:44.200210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:44.200176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:44.330107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:44.330071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:44.430228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:44.430198 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:47.759268 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:47.759233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:50.020194 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:50.020160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:50.091201 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:50.091169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:51.464544 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:51.464499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:52.020262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:52.020222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:52.124471 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:52.124424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:52.224145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:52.224110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:54.432830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:54.432793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:55.142313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:55.142272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:55.233844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:55.233808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:56.009190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:56.009158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:56.722810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:56.722776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:34:56.859946 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:56.859911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:57.786713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:57.786675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:58.732211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:58.732180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:59.733783 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.733743 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:34:59.737852 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.737832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.740994 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.740975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2d4fab-serving-cert\"" Sep 15 21:34:59.751651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.751626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:34:59.770030 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.770007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:59.862866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.862833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck2h\" (UniqueName: \"kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.863050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.862901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.863103 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.863074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.863151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.863121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.868824 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.868795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:59.964713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.964677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.964713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.964712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.964933 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.964776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck2h\" (UniqueName: \"kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.964933 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.964827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.965450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.965424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.967302 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.967276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.967411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.967339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:34:59.969018 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.968990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:34:59.973995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:34:59.973976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck2h\" (UniqueName: \"kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h\") pod \"maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:35:00.047875 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.047799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:35:00.339298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.339273 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:35:00.340579 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:35:00.340549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646e5395_813e_4dd1_8874_0100952e1d08.slice/crio-534ed8e947fd426874810a77756eb958836d223cbc64cd80cd797f19c4ccef45 WatchSource:0}: Error finding container 534ed8e947fd426874810a77756eb958836d223cbc64cd80cd797f19c4ccef45: Status 404 returned error can't find the container with id 534ed8e947fd426874810a77756eb958836d223cbc64cd80cd797f19c4ccef45 Sep 15 21:35:00.341990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.341971 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:35:00.427851 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.427813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:00.500172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.500142 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" event={"ID":"646e5395-813e-4dd1-8874-0100952e1d08","Type":"ContainerStarted","Data":"534ed8e947fd426874810a77756eb958836d223cbc64cd80cd797f19c4ccef45"} Sep 15 21:35:00.525198 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:00.525170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:01.507265 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:01.507230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" event={"ID":"646e5395-813e-4dd1-8874-0100952e1d08","Type":"ContainerStarted","Data":"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10"} Sep 15 21:35:01.507765 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:01.507430 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:35:01.543438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:01.543392 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" podStartSLOduration=2.1473886269999998 podStartE2EDuration="2.543374757s" totalImagesPullingTime="395.98613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:01.543327413 +0000 UTC m=+1149.767950416" watchObservedRunningTime="2026-09-15 21:35:01.543374757 +0000 UTC m=+1149.767997761" Sep 15 21:35:07.518187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:07.518155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:35:11.468002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:11.467966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:16.182865 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:16.182784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:17.053301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:17.053245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:18.078174 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:18.078116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:19.009217 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:19.009180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:19.951926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:19.951891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:20.945344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:20.945302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:21.778586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:21.778556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:22.533582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:22.533547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:23.381056 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:23.380994 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:23.619633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:23.619606 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:35:24.088729 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:24.088682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:24.466507 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:24.466399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:24.963152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:24.963120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.161160 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.161128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.396394 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.396360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.496895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.496777 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.778363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.778279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.883042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.883010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:25.976852 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:25.976820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:26.333128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:26.333092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:26.434540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:26.434511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:26.745277 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:26.745193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:27.447127 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:27.447099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:27.527108 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:27.527079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:27.631571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:27.631536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:27.743390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:27.743303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:28.111942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:28.111914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:28.228081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:28.228038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:28.974313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:28.974271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:29.076789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:29.076762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:29.175739 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:29.175707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:29.853229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:29.853199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:30.653565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:30.653537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:31.359951 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:31.359918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:32.222571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:32.222537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:32.396699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:32.396665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:33.261924 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:33.261884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:33.509347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:33.509314 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:33.601995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:33.601946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:33.879895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:33.879818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:34.267430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:34.267352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:34.370678 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:34.370651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:34.836226 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:34.836192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:36.178712 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:36.178669 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:53.720893 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:53.720856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:35:56.194763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:56.194722 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:56.922929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:56.922894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:57.763854 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:57.763787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:58.591739 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:58.591710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:59.049193 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.049112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:59.322680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.322598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:35:59.807212 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.807177 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-3bc0e7"] Sep 15 21:35:59.812975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.812957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:35:59.817472 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.817430 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\"" Sep 15 21:35:59.823797 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.823777 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-3bc0e7"] Sep 15 21:35:59.932785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:35:59.932753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq6n\" (UniqueName: \"kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n\") pod \"test-curl-7751-3bc0e7\" (UID: \"46ef53b1-4576-4947-891c-21b9a4f792a8\") " pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:00.034317 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.034282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq6n\" (UniqueName: \"kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n\") pod \"test-curl-7751-3bc0e7\" (UID: \"46ef53b1-4576-4947-891c-21b9a4f792a8\") " pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:00.042878 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.042848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq6n\" (UniqueName: \"kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n\") pod \"test-curl-7751-3bc0e7\" (UID: \"46ef53b1-4576-4947-891c-21b9a4f792a8\") " pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:00.121861 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.121836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:00.274207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.274180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:00.391581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.391558 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-3bc0e7"] Sep 15 21:36:00.393958 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:36:00.393929 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ef53b1_4576_4947_891c_21b9a4f792a8.slice/crio-7aa0cd47b3a13648b0d3b8142b47ba0960bde712a05ccb2075284407f43bd6eb WatchSource:0}: Error finding container 7aa0cd47b3a13648b0d3b8142b47ba0960bde712a05ccb2075284407f43bd6eb: Status 404 returned error can't find the container with id 7aa0cd47b3a13648b0d3b8142b47ba0960bde712a05ccb2075284407f43bd6eb Sep 15 21:36:00.821842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.821755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-3bc0e7" event={"ID":"46ef53b1-4576-4947-891c-21b9a4f792a8","Type":"ContainerStarted","Data":"31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a"} Sep 15 21:36:00.821842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.821796 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-3bc0e7" event={"ID":"46ef53b1-4576-4947-891c-21b9a4f792a8","Type":"ContainerStarted","Data":"7aa0cd47b3a13648b0d3b8142b47ba0960bde712a05ccb2075284407f43bd6eb"} Sep 15 21:36:00.838947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:00.838901 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-3bc0e7" podStartSLOduration=1.838887047 podStartE2EDuration="1.838887047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:00.836648323 +0000 UTC m=+1209.061271327" watchObservedRunningTime="2026-09-15 21:36:00.838887047 +0000 UTC m=+1209.063510050" Sep 15 21:36:01.009969 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:01.009940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:01.400238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:01.400205 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-3bc0e7"] Sep 15 21:36:01.408031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:01.408003 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-3bc0e7"] Sep 15 21:36:01.827082 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:01.827050 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-3bc0e7" container="test-curl-7751-3bc0e7" Sep 15 21:36:02.831786 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:02.831745 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-3bc0e7" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" containerName="test-curl-7751-3bc0e7" containerID="cri-o://31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a" gracePeriod=2 Sep 15 21:36:04.945737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.945707 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-529dcb"] Sep 15 21:36:04.946091 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.946078 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" containerName="test-curl-7751-3bc0e7" Sep 15 21:36:04.946138 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.946092 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" containerName="test-curl-7751-3bc0e7" Sep 15 21:36:04.946187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.946177 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" containerName="test-curl-7751-3bc0e7" Sep 15 21:36:04.949354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.949331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:04.959745 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:04.959593 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-529dcb"] Sep 15 21:36:05.023967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.023946 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:05.087144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.087082 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rq6n\" (UniqueName: \"kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n\") pod \"46ef53b1-4576-4947-891c-21b9a4f792a8\" (UID: \"46ef53b1-4576-4947-891c-21b9a4f792a8\") " Sep 15 21:36:05.087313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.087296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jj6\" (UniqueName: \"kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6\") pod \"test-curl-7751-529dcb\" (UID: \"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4\") " pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:05.089200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.089173 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n" (OuterVolumeSpecName: "kube-api-access-9rq6n") pod "46ef53b1-4576-4947-891c-21b9a4f792a8" (UID: "46ef53b1-4576-4947-891c-21b9a4f792a8"). InnerVolumeSpecName "kube-api-access-9rq6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:05.188001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.187963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jj6\" (UniqueName: \"kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6\") pod \"test-curl-7751-529dcb\" (UID: \"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4\") " pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:05.188174 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.188031 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rq6n\" (UniqueName: \"kubernetes.io/projected/46ef53b1-4576-4947-891c-21b9a4f792a8-kube-api-access-9rq6n\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:05.199153 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.199123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jj6\" (UniqueName: \"kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6\") pod \"test-curl-7751-529dcb\" (UID: \"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4\") " pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:05.267248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.267216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:05.537120 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.537095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-529dcb"] Sep 15 21:36:05.538829 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:36:05.538778 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf0c6b6a_c2a0_4eec_9eb2_b42eddc348a4.slice/crio-393eb7b3910b291b579793e65e7e67f8afcc9c602a0abdd63b63fba44ec6b71d WatchSource:0}: Error finding container 393eb7b3910b291b579793e65e7e67f8afcc9c602a0abdd63b63fba44ec6b71d: Status 404 returned error can't find the container with id 393eb7b3910b291b579793e65e7e67f8afcc9c602a0abdd63b63fba44ec6b71d Sep 15 21:36:05.857152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.857121 2559 generic.go:356] "Generic (PLEG): container finished" podID="46ef53b1-4576-4947-891c-21b9a4f792a8" containerID="31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a" exitCode=137 Sep 15 21:36:05.857322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.857167 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:05.857322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.857260 2559 scope.go:117] "RemoveContainer" containerID="31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a" Sep 15 21:36:05.859971 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.859941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-529dcb" event={"ID":"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4","Type":"ContainerStarted","Data":"c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc"} Sep 15 21:36:05.860103 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.859982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-529dcb" event={"ID":"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4","Type":"ContainerStarted","Data":"393eb7b3910b291b579793e65e7e67f8afcc9c602a0abdd63b63fba44ec6b71d"} Sep 15 21:36:05.868929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.868911 2559 scope.go:117] "RemoveContainer" containerID="31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a" Sep 15 21:36:05.869232 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:05.869212 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a\": container with ID starting with 31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a not found: ID does not exist" containerID="31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a" Sep 15 21:36:05.869313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.869239 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a"} err="failed to get container status \"31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a\": rpc error: code = NotFound desc = could not find container \"31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a\": container with ID starting with 31dfa9dc382ec8edeb7b56d9cf08f8174479e699ad05676f2e883b2d8836ed0a not found: ID does not exist" Sep 15 21:36:05.874782 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:05.874733 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-529dcb" podStartSLOduration=1.874716296 podStartE2EDuration="1.874716296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:05.874440579 +0000 UTC m=+1214.099063620" watchObservedRunningTime="2026-09-15 21:36:05.874716296 +0000 UTC m=+1214.099339302" Sep 15 21:36:05.876689 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:05.876659 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7751-3bc0e7\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" pod="opendatahub/test-curl-7751-3bc0e7" Sep 15 21:36:06.290125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:06.290082 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ef53b1-4576-4947-891c-21b9a4f792a8" path="/var/lib/kubelet/pods/46ef53b1-4576-4947-891c-21b9a4f792a8/volumes" Sep 15 21:36:06.446966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:06.446933 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-529dcb"] Sep 15 21:36:06.451337 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:06.451308 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-529dcb"] Sep 15 21:36:06.866643 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:06.866612 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-529dcb" container="test-curl-7751-529dcb" Sep 15 21:36:07.023147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.023113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:07.313153 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.313069 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:36:07.313706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.313657 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" containerName="test-curl-7751-529dcb" Sep 15 21:36:07.313706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.313678 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" containerName="test-curl-7751-529dcb" Sep 15 21:36:07.313899 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.313812 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" containerName="test-curl-7751-529dcb" Sep 15 21:36:07.318271 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.318249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.320846 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.320828 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-56820a3c-openshift-default-dockercfg-5t8br\"" Sep 15 21:36:07.330851 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.330831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:36:07.337722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.337699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:07.408529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.408988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.408876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fmls\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fmls\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.510966 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.510926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.511093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.511070 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.511162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.511147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.511330 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.511309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.512963 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.512939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.513271 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.513249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.519791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.519771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.519992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.519973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fmls\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls\") pod \"e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.628954 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.628924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:07.856224 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.856191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:07.872095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.872061 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-529dcb" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" containerName="test-curl-7751-529dcb" containerID="cri-o://c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc" gracePeriod=2 Sep 15 21:36:07.874429 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:07.874407 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7751-529dcb\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:07.911552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.911519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:36:07.914433 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:36:07.914400 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd73aed_e40b_4d5a_8cfa_91f39b8c43cb.slice/crio-fd3c79b6a07bd21242ef37edfea5f41fe71b67b34da7b9061736ff25ac102f72 WatchSource:0}: Error finding container fd3c79b6a07bd21242ef37edfea5f41fe71b67b34da7b9061736ff25ac102f72: Status 404 returned error can't find the container with id fd3c79b6a07bd21242ef37edfea5f41fe71b67b34da7b9061736ff25ac102f72 Sep 15 21:36:07.917070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.917035 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:07.917166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.917101 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:07.917166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:07.917131 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:08.409810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:08.409778 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:08.539223 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:08.539193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:08.878192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:08.878158 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerStarted","Data":"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91"} Sep 15 21:36:08.878348 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:08.878211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerStarted","Data":"fd3c79b6a07bd21242ef37edfea5f41fe71b67b34da7b9061736ff25ac102f72"} Sep 15 21:36:08.901651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:08.901601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podStartSLOduration=1.9015847369999999 podStartE2EDuration="1.901584737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:08.899350787 +0000 UTC m=+1217.123973788" watchObservedRunningTime="2026-09-15 21:36:08.901584737 +0000 UTC m=+1217.126207738" Sep 15 21:36:09.629414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:09.629378 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:09.634847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:09.634823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:09.883530 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:09.883470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:09.884253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:09.884231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:09.978310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:09.978280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:10.075776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.075751 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:10.122346 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.122316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:10.241167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.241083 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6jj6\" (UniqueName: \"kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6\") pod \"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4\" (UID: \"bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4\") " Sep 15 21:36:10.243057 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.243035 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6" (OuterVolumeSpecName: "kube-api-access-c6jj6") pod "bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" (UID: "bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4"). InnerVolumeSpecName "kube-api-access-c6jj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:10.290444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.290415 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" path="/var/lib/kubelet/pods/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4/volumes" Sep 15 21:36:10.342738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.342715 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6jj6\" (UniqueName: \"kubernetes.io/projected/bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4-kube-api-access-c6jj6\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:10.359204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.359176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:10.854630 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.854596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:10.889749 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.889714 2559 generic.go:356] "Generic (PLEG): container finished" podID="bf0c6b6a-c2a0-4eec-9eb2-b42eddc348a4" containerID="c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc" exitCode=137 Sep 15 21:36:10.889898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.889759 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-529dcb" Sep 15 21:36:10.889898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.889809 2559 scope.go:117] "RemoveContainer" containerID="c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc" Sep 15 21:36:10.898269 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.898249 2559 scope.go:117] "RemoveContainer" containerID="c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc" Sep 15 21:36:10.898543 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:36:10.898523 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc\": container with ID starting with c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc not found: ID does not exist" containerID="c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc" Sep 15 21:36:10.898594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.898552 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc"} err="failed to get container status \"c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc\": rpc error: code = NotFound desc = could not find container \"c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc\": container with ID starting with c2b894ed70f374571cb48ac4ada8f81e0e9f24cbcc66357b73e5554cc8c40bfc not found: ID does not exist" Sep 15 21:36:10.950134 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:10.950100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:11.821523 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:11.821478 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:12.734736 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:12.734702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:13.643257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:13.643227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:14.548325 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:14.548295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:15.453155 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:15.453122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:15.830211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:15.830185 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:16.545473 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.545437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:16.578491 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.577496 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:36:16.586277 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.586246 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.593198 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.593171 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-56820a3c-serving-cert\"" Sep 15 21:36:16.602138 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.602109 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:36:16.691742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.691712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:16.703849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.703827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.703921 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.703859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.703921 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.703891 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4llq\" (UniqueName: \"kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.704001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.703985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.805044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.804974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.805148 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.805045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.805148 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.805068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.805148 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.805098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4llq\" (UniqueName: \"kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.805791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.805768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.807489 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.807448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.807593 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.807537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:16.807650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.807625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.814277 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.814260 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4llq\" (UniqueName: \"kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq\") pod \"maas-api-e2e-mt-56820a3c-d846687ff-f5552\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:16.905397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:16.905362 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:17.255918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.255892 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:36:17.257669 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:36:17.257629 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5282e8b_7550_410a_b49d_371d585a8cc5.slice/crio-81fa95c6d492632243caea38a038dd9b7718dd562e16a036b92d995fbe6574f7 WatchSource:0}: Error finding container 81fa95c6d492632243caea38a038dd9b7718dd562e16a036b92d995fbe6574f7: Status 404 returned error can't find the container with id 81fa95c6d492632243caea38a038dd9b7718dd562e16a036b92d995fbe6574f7 Sep 15 21:36:17.266344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.266303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:17.456301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.456273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:17.929827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.929785 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" event={"ID":"c5282e8b-7550-410a-b49d-371d585a8cc5","Type":"ContainerStarted","Data":"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c"} Sep 15 21:36:17.930311 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.929842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" event={"ID":"c5282e8b-7550-410a-b49d-371d585a8cc5","Type":"ContainerStarted","Data":"81fa95c6d492632243caea38a038dd9b7718dd562e16a036b92d995fbe6574f7"} Sep 15 21:36:17.930311 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.929868 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:17.952054 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:17.952009 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" podStartSLOduration=1.476899899 podStartE2EDuration="1.951993477s" totalImagesPullingTime="475.093578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:17.949356678 +0000 UTC m=+1226.173979687" watchObservedRunningTime="2026-09-15 21:36:17.951993477 +0000 UTC m=+1226.176616480" Sep 15 21:36:18.330873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:18.330842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:36:22.128907 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:22.128875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:23.041760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:23.041724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:23.796749 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:23.796713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:23.941107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:23.941079 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:36:24.769330 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:24.769295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:25.459626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:25.459589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:26.164169 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:26.164138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:26.936565 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:26.936527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.005284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.005255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.077843 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.077785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.207365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.207258 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.618360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.618327 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.885515 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.885405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:27.981831 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:27.981790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:28.086977 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:28.086932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:28.634004 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:28.633969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:29.657858 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:29.657823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:29.761177 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:29.761146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:29.861758 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:29.861724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:29.958190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:29.958093 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:30.550218 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:30.550184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:30.651250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:30.651218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:31.290851 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:31.290818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:32.060188 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:32.060157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:32.968358 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:32.968322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:33.779384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:33.779351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:34.501979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:34.501918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:35.373189 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:35.373152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.005776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.005741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.105105 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.105074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.207984 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.207956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.302234 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.302154 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.538002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.537970 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:36.650579 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:36.650414 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:37.599589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:37.599557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:38.559006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:38.558972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:39.463225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:39.463190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:40.259945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:40.259871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:40.992238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:40.992208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:41.878380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:41.878317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:42.690264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:42.690225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:42.790933 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:42.790908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:42.955962 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:42.955879 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:36:42.956497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:42.956429 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" containerID="cri-o://b785abe72ac9876df352a0f8d4d5eee835b7ce4822b30480dee3eaa416382d18" gracePeriod=30 Sep 15 21:36:42.990865 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:42.990835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:44.390658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:44.390624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:44.985133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:44.985043 2559 patch_prober.go:28] interesting pod/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:36:44.985446 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:44.985419 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:36:45.566884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:45.566846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:45.941961 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:45.941923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:46.798622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:46.798587 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:47.663480 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:47.663433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:48.110800 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.110762 2559 generic.go:356] "Generic (PLEG): container finished" podID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerID="b785abe72ac9876df352a0f8d4d5eee835b7ce4822b30480dee3eaa416382d18" exitCode=0 Sep 15 21:36:48.111187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.110840 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerDied","Data":"b785abe72ac9876df352a0f8d4d5eee835b7ce4822b30480dee3eaa416382d18"} Sep 15 21:36:48.111187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.110917 2559 scope.go:117] "RemoveContainer" containerID="febdcedd8fef371fb2a2808a99c7fc99c20b7c26352fd67b081970c3c6a41006" Sep 15 21:36:48.215193 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.215165 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333142 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333194 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333225 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ch45\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333377 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333402 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333409 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333418 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333620 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333667 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket\") pod \"2d53c5aa-c553-4c19-9437-cd27274ff201\" (UID: \"2d53c5aa-c553-4c19-9437-cd27274ff201\") " Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333676 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333681 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:48.333880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333854 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.333986 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.334144 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.334163 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.334177 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d53c5aa-c553-4c19-9437-cd27274ff201-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.334189 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.335033 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.334203 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.335617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.335587 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:36:48.335813 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.335790 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:48.336024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.336006 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:36:48.336073 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.336018 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45" (OuterVolumeSpecName: "kube-api-access-6ch45") pod "2d53c5aa-c553-4c19-9437-cd27274ff201" (UID: "2d53c5aa-c553-4c19-9437-cd27274ff201"). InnerVolumeSpecName "kube-api-access-6ch45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:48.435349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.435325 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ch45\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-kube-api-access-6ch45\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.435349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.435350 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.435548 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.435363 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.435548 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.435375 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d53c5aa-c553-4c19-9437-cd27274ff201-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:48.517396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:48.517368 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:49.118006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.117977 2559 generic.go:356] "Generic (PLEG): container finished" podID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerID="27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91" exitCode=0 Sep 15 21:36:49.118394 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.118053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerDied","Data":"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91"} Sep 15 21:36:49.118564 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.118544 2559 scope.go:117] "RemoveContainer" containerID="27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91" Sep 15 21:36:49.119620 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119594 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" Sep 15 21:36:49.119620 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7" event={"ID":"2d53c5aa-c553-4c19-9437-cd27274ff201","Type":"ContainerDied","Data":"eeeb99f7d3de8a1fc91fe8c0245dd40001648884655a0c58f2eeb01e1bf649d3"} Sep 15 21:36:49.119733 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119653 2559 scope.go:117] "RemoveContainer" containerID="b785abe72ac9876df352a0f8d4d5eee835b7ce4822b30480dee3eaa416382d18" Sep 15 21:36:49.119839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119811 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:49.119904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119887 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:49.119949 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.119927 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:36:49.159770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.159726 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:36:49.164374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.164350 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4bc420-openshift-default-7b8455794d-867z7"] Sep 15 21:36:49.263993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:49.263867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:50.127356 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:50.127303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerStarted","Data":"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c"} Sep 15 21:36:50.293376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:50.293339 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" path="/var/lib/kubelet/pods/2d53c5aa-c553-4c19-9437-cd27274ff201/volumes" Sep 15 21:36:50.629667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:50.629631 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:50.630901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:50.630874 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 15 21:36:50.631021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:50.630927 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 15 21:36:51.629402 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:51.629368 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 15 21:36:51.629809 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:51.629423 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 15 21:36:52.226014 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:52.225987 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:36:52.281996 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:52.281927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:52.629773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:52.629736 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 15 21:36:52.630242 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:52.629813 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 15 21:36:53.629635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:53.629600 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 15 21:36:53.629811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:53.629655 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 15 21:36:53.968531 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:53.966660 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:36:53.968531 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:53.966958 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" podUID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" containerName="maas-api" containerID="cri-o://59bee60ce37707a0416d71b8edf665e0868c3ef3f14c4fa7f1689d13f6be7cbd" gracePeriod=30 Sep 15 21:36:54.003800 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.003759 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.081965 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.081894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.152191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.152158 2559 generic.go:356] "Generic (PLEG): container finished" podID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" containerID="59bee60ce37707a0416d71b8edf665e0868c3ef3f14c4fa7f1689d13f6be7cbd" exitCode=0 Sep 15 21:36:54.152314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.152240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" event={"ID":"4f3d2192-4f4e-44d3-88a7-ab9a5964687c","Type":"ContainerDied","Data":"59bee60ce37707a0416d71b8edf665e0868c3ef3f14c4fa7f1689d13f6be7cbd"} Sep 15 21:36:54.187487 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.187007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.264070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.263918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.313262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.313231 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:36:54.403985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.403956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca\") pod \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " Sep 15 21:36:54.404144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.404000 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls\") pod \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " Sep 15 21:36:54.404144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.404022 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs\") pod \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " Sep 15 21:36:54.404144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.404099 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpnls\" (UniqueName: \"kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls\") pod \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\" (UID: \"4f3d2192-4f4e-44d3-88a7-ab9a5964687c\") " Sep 15 21:36:54.404589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.404564 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f3d2192-4f4e-44d3-88a7-ab9a5964687c" (UID: "4f3d2192-4f4e-44d3-88a7-ab9a5964687c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:36:54.406383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.406355 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f3d2192-4f4e-44d3-88a7-ab9a5964687c" (UID: "4f3d2192-4f4e-44d3-88a7-ab9a5964687c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:36:54.406383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.406374 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f3d2192-4f4e-44d3-88a7-ab9a5964687c" (UID: "4f3d2192-4f4e-44d3-88a7-ab9a5964687c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:36:54.406532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.406367 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls" (OuterVolumeSpecName: "kube-api-access-rpnls") pod "4f3d2192-4f4e-44d3-88a7-ab9a5964687c" (UID: "4f3d2192-4f4e-44d3-88a7-ab9a5964687c"). InnerVolumeSpecName "kube-api-access-rpnls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:36:54.504941 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.504909 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.504941 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.504939 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.504941 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.504949 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.505150 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.504957 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnls\" (UniqueName: \"kubernetes.io/projected/4f3d2192-4f4e-44d3-88a7-ab9a5964687c-kube-api-access-rpnls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:36:54.591764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.591727 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.630334 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.630306 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 15 21:36:54.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.630362 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 15 21:36:54.654032 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.654004 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:54.883610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:54.883510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:55.160748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.160658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" event={"ID":"4f3d2192-4f4e-44d3-88a7-ab9a5964687c","Type":"ContainerDied","Data":"b3152dd473366c6033e7bdd938637b1b48abff3a591f7902ac8e9a0756c7b9fb"} Sep 15 21:36:55.160748 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.160735 2559 scope.go:117] "RemoveContainer" containerID="59bee60ce37707a0416d71b8edf665e0868c3ef3f14c4fa7f1689d13f6be7cbd" Sep 15 21:36:55.160969 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.160909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865" Sep 15 21:36:55.190062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.190031 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:36:55.194487 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.194443 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb7867-6cb5ccdc7b-zx865"] Sep 15 21:36:55.468520 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.468411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:55.634194 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.634168 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:55.635072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:55.635055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:36:56.289929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:56.289898 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" path="/var/lib/kubelet/pods/4f3d2192-4f4e-44d3-88a7-ab9a5964687c/volumes" Sep 15 21:36:56.380017 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:56.379986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:57.266676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:57.266646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:57.646312 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:57.646280 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:36:58.075122 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:58.075040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:58.932040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:58.932009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:36:59.670837 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:36:59.670801 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:06.567816 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:06.567780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:07.245130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.245075 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" podUID="ae008835-baac-403f-9eb4-4784935f0b8d" containerName="main" containerID="cri-o://90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a" gracePeriod=45 Sep 15 21:37:07.511368 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.511345 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:37:07.621402 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621363 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621487 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnq42\" (UniqueName: \"kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621538 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621553 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621578 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache\") pod \"ae008835-baac-403f-9eb4-4784935f0b8d\" (UID: \"ae008835-baac-403f-9eb4-4784935f0b8d\") " Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621620 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:07.621839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home" (OuterVolumeSpecName: "home") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:07.622098 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621897 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:07.622098 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621895 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:07.622098 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.621916 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:07.623621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.623596 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:37:07.623709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.623685 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42" (OuterVolumeSpecName: "kube-api-access-mnq42") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "kube-api-access-mnq42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:37:07.623758 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.623716 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm" (OuterVolumeSpecName: "dshm") pod "ae008835-baac-403f-9eb4-4784935f0b8d" (UID: "ae008835-baac-403f-9eb4-4784935f0b8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:07.722848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.722796 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae008835-baac-403f-9eb4-4784935f0b8d-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:07.722848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.722839 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:07.722848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.722851 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae008835-baac-403f-9eb4-4784935f0b8d-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:07.723089 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:07.722863 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnq42\" (UniqueName: \"kubernetes.io/projected/ae008835-baac-403f-9eb4-4784935f0b8d-kube-api-access-mnq42\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:08.229609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.229575 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae008835-baac-403f-9eb4-4784935f0b8d" containerID="90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a" exitCode=0 Sep 15 21:37:08.229808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.229618 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" Sep 15 21:37:08.229808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.229646 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" event={"ID":"ae008835-baac-403f-9eb4-4784935f0b8d","Type":"ContainerDied","Data":"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a"} Sep 15 21:37:08.229808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.229694 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr" event={"ID":"ae008835-baac-403f-9eb4-4784935f0b8d","Type":"ContainerDied","Data":"a450e6485233b4c9b0131496984e76764184319284269bb1b8473b298e150f09"} Sep 15 21:37:08.229808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.229716 2559 scope.go:117] "RemoveContainer" containerID="90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a" Sep 15 21:37:08.239386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.239367 2559 scope.go:117] "RemoveContainer" containerID="90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a" Sep 15 21:37:08.239721 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:37:08.239703 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a\": container with ID starting with 90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a not found: ID does not exist" containerID="90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a" Sep 15 21:37:08.239804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.239727 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a"} err="failed to get container status \"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a\": rpc error: code = NotFound desc = could not find container \"90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a\": container with ID starting with 90a76a6b439f4d05175d72d7b0ca4a96a0eb85b37e4bbedc1de8933f7f76f53a not found: ID does not exist" Sep 15 21:37:08.253084 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.253047 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:37:08.255091 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.255070 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bb7867/test-model-bb7867-kserve-7c748c49c-bx6lr"] Sep 15 21:37:08.289868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:08.289847 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae008835-baac-403f-9eb4-4784935f0b8d" path="/var/lib/kubelet/pods/ae008835-baac-403f-9eb4-4784935f0b8d/volumes" Sep 15 21:37:13.381960 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:13.381930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:13.620304 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:13.620271 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:37:13.620628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:13.620548 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" containerID="cri-o://51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e" gracePeriod=30 Sep 15 21:37:13.651264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:13.651188 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:14.172211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:14.172174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:14.508217 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:14.508124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:14.935356 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:14.935329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:16.637101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:16.637068 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:37:17.199301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:17.199269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:17.589551 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:17.589515 2559 patch_prober.go:28] interesting pod/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:37:17.589690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:17.589581 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:37:17.752385 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:17.752351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:17.861522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:17.861414 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:18.356152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.356118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:18.880190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.880167 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:37:18.936648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936584 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.936648 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936635 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.936844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936667 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.936844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.936844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxmm8\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.937005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.937005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:18.937005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936940 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:18.937005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.937005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.936995 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.937267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937028 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs\") pod \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\" (UID: \"cebeead4-7eb7-4f91-8da6-aeaff08ce88d\") " Sep 15 21:37:18.937336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937300 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:18.937392 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937353 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:37:18.937573 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937549 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:18.937664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937619 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:18.937664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937640 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:18.937664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937655 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:18.937838 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.937669 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:18.939047 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.939012 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:37:18.939151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.939042 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8" (OuterVolumeSpecName: "kube-api-access-cxmm8") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "kube-api-access-cxmm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:37:18.939216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.939159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:37:18.939216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:18.939171 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cebeead4-7eb7-4f91-8da6-aeaff08ce88d" (UID: "cebeead4-7eb7-4f91-8da6-aeaff08ce88d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:37:19.038811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.038787 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:19.038811 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.038811 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:19.038998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.038819 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:19.038998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.038831 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxmm8\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-kube-api-access-cxmm8\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:19.038998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.038840 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebeead4-7eb7-4f91-8da6-aeaff08ce88d-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:37:19.271512 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.271410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:19.298647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.298619 2559 generic.go:356] "Generic (PLEG): container finished" podID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerID="51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e" exitCode=0 Sep 15 21:37:19.298769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.298670 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" Sep 15 21:37:19.298769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.298696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerDied","Data":"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e"} Sep 15 21:37:19.298769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.298741 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9" event={"ID":"cebeead4-7eb7-4f91-8da6-aeaff08ce88d","Type":"ContainerDied","Data":"25d997e76c76a39539378a166618275cc0d0672d90ff6cbaf8b431269f23b8d3"} Sep 15 21:37:19.298769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.298762 2559 scope.go:117] "RemoveContainer" containerID="51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e" Sep 15 21:37:19.308264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.308248 2559 scope.go:117] "RemoveContainer" containerID="318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626" Sep 15 21:37:19.316824 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.316807 2559 scope.go:117] "RemoveContainer" containerID="51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e" Sep 15 21:37:19.317076 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:37:19.317057 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e\": container with ID starting with 51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e not found: ID does not exist" containerID="51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e" Sep 15 21:37:19.317129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.317084 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e"} err="failed to get container status \"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e\": rpc error: code = NotFound desc = could not find container \"51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e\": container with ID starting with 51755d801a98751768842653bf28fefb37f5bfcc390b15f7f643abe31724642e not found: ID does not exist" Sep 15 21:37:19.317129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.317103 2559 scope.go:117] "RemoveContainer" containerID="318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626" Sep 15 21:37:19.317332 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:37:19.317313 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626\": container with ID starting with 318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626 not found: ID does not exist" containerID="318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626" Sep 15 21:37:19.317382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.317337 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626"} err="failed to get container status \"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626\": rpc error: code = NotFound desc = could not find container \"318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626\": container with ID starting with 318d3c1ec66e001cae0ab9ebe22ef8f8b97d2f2309f90c64d44513b5580ff626 not found: ID does not exist" Sep 15 21:37:19.326951 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.324122 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:37:19.327668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:19.327642 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bb7867-openshift-default-7d9d499c4d-875s9"] Sep 15 21:37:20.035450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:20.035421 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:20.291093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:20.291021 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" path="/var/lib/kubelet/pods/cebeead4-7eb7-4f91-8da6-aeaff08ce88d/volumes" Sep 15 21:37:20.904235 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:20.904197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:21.660384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:21.660344 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:22.577170 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577137 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:37:22.577730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577705 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:22.577730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577729 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577753 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" containerName="maas-api" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577761 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" containerName="maas-api" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577775 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577780 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577795 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae008835-baac-403f-9eb4-4784935f0b8d" containerName="main" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577801 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae008835-baac-403f-9eb4-4784935f0b8d" containerName="main" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577814 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577818 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577889 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae008835-baac-403f-9eb4-4784935f0b8d" containerName="main" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577901 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:22.577905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577910 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.578360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577920 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f3d2192-4f4e-44d3-88a7-ab9a5964687c" containerName="maas-api" Sep 15 21:37:22.578360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.577929 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebeead4-7eb7-4f91-8da6-aeaff08ce88d" containerName="istio-proxy" Sep 15 21:37:22.581211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.581189 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.584829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.584803 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-59e243-serving-cert\"" Sep 15 21:37:22.601068 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.601048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:37:22.648405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.648375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:22.674369 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.674337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcm2\" (UniqueName: \"kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.674780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.674538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.674780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.674608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.674780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.674668 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.773416 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.773362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:22.775676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.775655 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.775749 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.775710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.775789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.775749 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.775829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.775814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcm2\" (UniqueName: \"kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.775960 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:37:22.775942 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-serving-cert: secret "maas-api-e2e-shared-a-w3-59e243-serving-cert" not found Sep 15 21:37:22.776025 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:37:22.776015 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls podName:07856ddb-f228-4c4a-9826-73f0550a32b5 nodeName:}" failed. No retries permitted until 2026-09-15 21:37:23.27600047 +0000 UTC m=+1291.500623451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls") pod "maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" (UID: "07856ddb-f228-4c4a-9826-73f0550a32b5") : secret "maas-api-e2e-shared-a-w3-59e243-serving-cert" not found Sep 15 21:37:22.776376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.776350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.778057 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.778040 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:22.784832 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:22.784813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcm2\" (UniqueName: \"kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:23.074251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.074168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:23.175671 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.175640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:23.270376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.270340 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:23.281702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.281674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:23.284002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.283972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:23.492625 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.492587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:23.768281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:23.768259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:37:23.770552 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:37:23.770523 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07856ddb_f228_4c4a_9826_73f0550a32b5.slice/crio-8220bac8b3620c1aa48e48415516d9b556614ee8d4d7abdb5cf2f56ee422a1d4 WatchSource:0}: Error finding container 8220bac8b3620c1aa48e48415516d9b556614ee8d4d7abdb5cf2f56ee422a1d4: Status 404 returned error can't find the container with id 8220bac8b3620c1aa48e48415516d9b556614ee8d4d7abdb5cf2f56ee422a1d4 Sep 15 21:37:24.327042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:24.327011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" event={"ID":"07856ddb-f228-4c4a-9826-73f0550a32b5","Type":"ContainerStarted","Data":"8220bac8b3620c1aa48e48415516d9b556614ee8d4d7abdb5cf2f56ee422a1d4"} Sep 15 21:37:25.333827 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:25.333792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" event={"ID":"07856ddb-f228-4c4a-9826-73f0550a32b5","Type":"ContainerStarted","Data":"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2"} Sep 15 21:37:25.334187 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:25.333901 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:25.365376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:25.365328 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" podStartSLOduration=2.744951239 podStartE2EDuration="3.365312023s" totalImagesPullingTime="620.360784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:25.363661799 +0000 UTC m=+1293.588284805" watchObservedRunningTime="2026-09-15 21:37:25.365312023 +0000 UTC m=+1293.589935026" Sep 15 21:37:29.706657 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.706622 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:37:29.707045 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.707031 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:29.707096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.707046 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:29.707128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.707117 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d53c5aa-c553-4c19-9437-cd27274ff201" containerName="istio-proxy" Sep 15 21:37:29.710291 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.710271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.713217 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.713194 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4a69fe\"/\"default-dockercfg-94zf9\"" Sep 15 21:37:29.714397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.714372 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4a69fe\"/\"noten-model-4a69fe-kserve-self-signed-certs\"" Sep 15 21:37:29.714533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.714412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4a69fe\"/\"kube-root-ca.crt\"" Sep 15 21:37:29.714533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.714497 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4a69fe\"/\"openshift-service-ca.crt\"" Sep 15 21:37:29.724002 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.723977 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:37:29.839090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.839222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.839222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmz6\" (UniqueName: \"kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.839222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.839333 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.839333 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.839285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.939960 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.939930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.939968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.939990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmz6\" (UniqueName: \"kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.940541 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.940491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.942204 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.942183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.942486 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.942448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:29.949087 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:29.949070 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmz6\" (UniqueName: \"kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6\") pod \"noten-model-4a69fe-kserve-689b94b5f5-k6xkm\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:30.019560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.019501 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:30.208979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.208952 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:30.295891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.295863 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:37:30.296939 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:37:30.296915 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d59160_bcc6_4de7_81af_658ad19a2c7f.slice/crio-05921bbacbf0527688f9185cf639b404b21cbb108bf6fd06ed197378c0cc9911 WatchSource:0}: Error finding container 05921bbacbf0527688f9185cf639b404b21cbb108bf6fd06ed197378c0cc9911: Status 404 returned error can't find the container with id 05921bbacbf0527688f9185cf639b404b21cbb108bf6fd06ed197378c0cc9911 Sep 15 21:37:30.360705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.360665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" event={"ID":"40d59160-bcc6-4de7-81af-658ad19a2c7f","Type":"ContainerStarted","Data":"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e"} Sep 15 21:37:30.360815 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.360717 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" event={"ID":"40d59160-bcc6-4de7-81af-658ad19a2c7f","Type":"ContainerStarted","Data":"05921bbacbf0527688f9185cf639b404b21cbb108bf6fd06ed197378c0cc9911"} Sep 15 21:37:30.379617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:30.379575 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" podStartSLOduration=1.379560179 podStartE2EDuration="1.379560179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:30.378811876 +0000 UTC m=+1298.603434881" watchObservedRunningTime="2026-09-15 21:37:30.379560179 +0000 UTC m=+1298.604183235" Sep 15 21:37:31.344064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:31.344032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:37:37.837419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:37.837382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:37:40.019822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:40.019781 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:40.020208 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:40.019836 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:40.032659 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:40.032630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:40.427740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:40.427711 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:37:45.746538 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:45.746502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:46.562148 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:46.562112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:47.561068 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:47.561035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:47.756817 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:47.756781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:48.080440 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:48.080402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:48.462955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:48.462875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:48.575114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:48.575085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:48.818576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:48.818489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:49.479194 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:49.479163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:50.264360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:50.264325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:51.227011 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:51.226974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:52.045212 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:52.045179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:52.145385 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:52.145353 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:52.945269 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:52.945236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:54.077970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:54.077937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:54.834199 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:54.834166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:55.773604 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:55.773569 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:56.543051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:56.543018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:57.298076 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:57.298040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:58.180034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.179998 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:37:58.184794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.184772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.188258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.188233 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-38a3a3-serving-cert\"" Sep 15 21:37:58.196698 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.196674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:37:58.211572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.211544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:58.309734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.309701 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv86l\" (UniqueName: \"kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.310110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.309741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.310110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.309801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.310110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.309821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.311270 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.311246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:58.410710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.410678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv86l\" (UniqueName: \"kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.410891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.410723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.410891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.410796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.410891 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.410831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.411526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.411499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:58.411649 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.411566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.413363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.413341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.413484 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.413390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.419530 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.419506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv86l\" (UniqueName: \"kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l\") pod \"maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.495437 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.495338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:58.774614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.774529 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:37:58.779801 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:37:58.779768 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b8124_8735_4f80_a2b5_545d1887e296.slice/crio-c7dba8dcf2c740167f27343dc9131b94cfeb9fbe4b1dfd30abc19e3fee400b4a WatchSource:0}: Error finding container c7dba8dcf2c740167f27343dc9131b94cfeb9fbe4b1dfd30abc19e3fee400b4a: Status 404 returned error can't find the container with id c7dba8dcf2c740167f27343dc9131b94cfeb9fbe4b1dfd30abc19e3fee400b4a Sep 15 21:37:58.864621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.864588 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:58.964114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:58.964077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:37:59.519943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:59.519857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" event={"ID":"959b8124-8735-4f80-a2b5-545d1887e296","Type":"ContainerStarted","Data":"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f"} Sep 15 21:37:59.519943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:59.519907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" event={"ID":"959b8124-8735-4f80-a2b5-545d1887e296","Type":"ContainerStarted","Data":"c7dba8dcf2c740167f27343dc9131b94cfeb9fbe4b1dfd30abc19e3fee400b4a"} Sep 15 21:37:59.520339 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:59.519964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:37:59.538780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:37:59.538729 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" podStartSLOduration=1.1189037609999999 podStartE2EDuration="1.538710641s" totalImagesPullingTime="419.80688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:59.538184128 +0000 UTC m=+1327.762807155" watchObservedRunningTime="2026-09-15 21:37:59.538710641 +0000 UTC m=+1327.763333646" Sep 15 21:38:03.364833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:03.364802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:03.595007 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:03.594976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:03.706059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:03.705978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:05.531247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:05.531218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:38:07.307184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.307150 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:38:07.310882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.310863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.314603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.314575 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56820a3c\"/\"default-dockercfg-xs78s\"" Sep 15 21:38:07.314603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.314581 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56820a3c\"/\"openshift-service-ca.crt\"" Sep 15 21:38:07.314785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.314616 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-56820a3c\"/\"kube-root-ca.crt\"" Sep 15 21:38:07.314785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.314663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-56820a3c\"/\"e2e-lifecycle-model-56820a3c-kserve-self-signed-certs\"" Sep 15 21:38:07.328990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.328967 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:38:07.393329 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.393497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.393497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.393497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88w5\" (UniqueName: \"kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.393658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.393658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.393576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494378 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88w5\" (UniqueName: \"kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494894 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494876 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.494945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.494923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.496717 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.496692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.496983 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.496964 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.503086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.503069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88w5\" (UniqueName: \"kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5\") pod \"e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.626258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.626222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:07.730389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.730358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:07.904212 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:07.904183 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:38:07.906347 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:38:07.906321 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod771e1482_23ea_482a_b2c2_ebe30381035a.slice/crio-4e26f7fdb842e0ee6a9159279c381eea8a13b9f2dded51043cd7d23894cdfa3f WatchSource:0}: Error finding container 4e26f7fdb842e0ee6a9159279c381eea8a13b9f2dded51043cd7d23894cdfa3f: Status 404 returned error can't find the container with id 4e26f7fdb842e0ee6a9159279c381eea8a13b9f2dded51043cd7d23894cdfa3f Sep 15 21:38:08.570686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:08.570651 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" event={"ID":"771e1482-23ea-482a-b2c2-ebe30381035a","Type":"ContainerStarted","Data":"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66"} Sep 15 21:38:08.570686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:08.570694 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" event={"ID":"771e1482-23ea-482a-b2c2-ebe30381035a","Type":"ContainerStarted","Data":"4e26f7fdb842e0ee6a9159279c381eea8a13b9f2dded51043cd7d23894cdfa3f"} Sep 15 21:38:08.594815 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:08.594770 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" podStartSLOduration=1.594755802 podStartE2EDuration="1.594755802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:08.593572241 +0000 UTC m=+1336.818195246" watchObservedRunningTime="2026-09-15 21:38:08.594755802 +0000 UTC m=+1336.819378800" Sep 15 21:38:08.682799 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:08.682772 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:38:08.730128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:08.730100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:15.532894 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:15.532816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:38:17.626328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:17.626298 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:17.626795 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:17.626345 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:17.638780 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:17.638756 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:18.638048 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:18.638017 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:38:20.745591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:20.745557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:21.578691 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:21.578657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:22.036216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:22.036117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:22.375231 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:22.375192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:23.389549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:23.389517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:23.701915 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:23.701834 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" podUID="40d59160-bcc6-4de7-81af-658ad19a2c7f" containerName="main" containerID="cri-o://88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e" gracePeriod=45 Sep 15 21:38:23.956241 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:23.956186 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:38:24.063731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063705 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.063895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063757 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.063895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.063895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.063895 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063883 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.064119 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.063905 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkmz6\" (UniqueName: \"kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6\") pod \"40d59160-bcc6-4de7-81af-658ad19a2c7f\" (UID: \"40d59160-bcc6-4de7-81af-658ad19a2c7f\") " Sep 15 21:38:24.064119 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.064005 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home" (OuterVolumeSpecName: "home") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:38:24.064247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.064170 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:38:24.064247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.064184 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:38:24.064247 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.064235 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.064367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.064250 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.066056 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.066028 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm" (OuterVolumeSpecName: "dshm") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:38:24.066184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.066090 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6" (OuterVolumeSpecName: "kube-api-access-jkmz6") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "kube-api-access-jkmz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:38:24.066184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.066145 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40d59160-bcc6-4de7-81af-658ad19a2c7f" (UID: "40d59160-bcc6-4de7-81af-658ad19a2c7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:38:24.164937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.164905 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.164937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.164931 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d59160-bcc6-4de7-81af-658ad19a2c7f-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.164937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.164942 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d59160-bcc6-4de7-81af-658ad19a2c7f-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.165141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.164950 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmz6\" (UniqueName: \"kubernetes.io/projected/40d59160-bcc6-4de7-81af-658ad19a2c7f-kube-api-access-jkmz6\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:38:24.170542 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.170509 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:24.669289 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.669252 2559 generic.go:356] "Generic (PLEG): container finished" podID="40d59160-bcc6-4de7-81af-658ad19a2c7f" containerID="88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e" exitCode=0 Sep 15 21:38:24.669815 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.669314 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" Sep 15 21:38:24.669920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.669888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" event={"ID":"40d59160-bcc6-4de7-81af-658ad19a2c7f","Type":"ContainerDied","Data":"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e"} Sep 15 21:38:24.670061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.669941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm" event={"ID":"40d59160-bcc6-4de7-81af-658ad19a2c7f","Type":"ContainerDied","Data":"05921bbacbf0527688f9185cf639b404b21cbb108bf6fd06ed197378c0cc9911"} Sep 15 21:38:24.670061 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.669964 2559 scope.go:117] "RemoveContainer" containerID="88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e" Sep 15 21:38:24.678610 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.678589 2559 scope.go:117] "RemoveContainer" containerID="88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e" Sep 15 21:38:24.678887 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:38:24.678864 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e\": container with ID starting with 88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e not found: ID does not exist" containerID="88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e" Sep 15 21:38:24.678952 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.678892 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e"} err="failed to get container status \"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e\": rpc error: code = NotFound desc = could not find container \"88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e\": container with ID starting with 88519f4c15d17074a9b68304988833c9a6405e9f506aa84a99e9fa7ea005267e not found: ID does not exist" Sep 15 21:38:24.686004 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.685978 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:38:24.689699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.689675 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4a69fe/noten-model-4a69fe-kserve-689b94b5f5-k6xkm"] Sep 15 21:38:24.906243 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:24.906213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:25.794079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:25.794041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:26.290471 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:26.290425 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d59160-bcc6-4de7-81af-658ad19a2c7f" path="/var/lib/kubelet/pods/40d59160-bcc6-4de7-81af-658ad19a2c7f/volumes" Sep 15 21:38:26.563432 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:26.563351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:30.206860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.206825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:30.523404 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.523329 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:38:30.523951 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.523933 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d59160-bcc6-4de7-81af-658ad19a2c7f" containerName="main" Sep 15 21:38:30.524004 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.523953 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d59160-bcc6-4de7-81af-658ad19a2c7f" containerName="main" Sep 15 21:38:30.524051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.524023 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d59160-bcc6-4de7-81af-658ad19a2c7f" containerName="main" Sep 15 21:38:30.528416 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.528397 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.530998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.530975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"e2e-conflict-b-473af5c1-kserve-self-signed-certs\"" Sep 15 21:38:30.539350 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.539328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:38:30.564280 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.564258 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:30.624812 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.624792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.624904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.624821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.624904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.624846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.624990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.624899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdch2\" (UniqueName: \"kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.624990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.624942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.625066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.625026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdch2\" (UniqueName: \"kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726903 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726963 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.726963 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.726931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.729184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.729163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.729347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.729330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.736934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.736911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdch2\" (UniqueName: \"kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2\") pod \"e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:30.838830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:30.838801 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:31.112283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:31.112255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:38:31.114434 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:38:31.114403 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c5d6e48_80dc_4aaf_b059_ef1301293b99.slice/crio-aa8af8123c50676d2873da6c93060fe04ca1c55ad74059f559c9c6b584075822 WatchSource:0}: Error finding container aa8af8123c50676d2873da6c93060fe04ca1c55ad74059f559c9c6b584075822: Status 404 returned error can't find the container with id aa8af8123c50676d2873da6c93060fe04ca1c55ad74059f559c9c6b584075822 Sep 15 21:38:31.713889 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:31.713855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" event={"ID":"9c5d6e48-80dc-4aaf-b059-ef1301293b99","Type":"ContainerStarted","Data":"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728"} Sep 15 21:38:31.713889 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:31.713899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" event={"ID":"9c5d6e48-80dc-4aaf-b059-ef1301293b99","Type":"ContainerStarted","Data":"aa8af8123c50676d2873da6c93060fe04ca1c55ad74059f559c9c6b584075822"} Sep 15 21:38:31.737407 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:31.737349 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" podStartSLOduration=1.737329377 podStartE2EDuration="1.737329377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:31.734933339 +0000 UTC m=+1359.959556343" watchObservedRunningTime="2026-09-15 21:38:31.737329377 +0000 UTC m=+1359.961952381" Sep 15 21:38:37.165982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:37.165947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:40.415857 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:40.415821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:40.839706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:40.839675 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:40.839916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:40.839723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:40.852519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:40.852499 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:41.790571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:41.790545 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:38:45.626495 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:45.626438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:38:48.530152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:38:48.530120 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:39:13.135957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:13.135925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:18.874173 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:18.874137 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:19.575667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:19.575635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:20.270149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:20.270112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:20.984728 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:20.984665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:21.868954 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:21.868917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:22.591336 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:22.591299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:22.802438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:22.802397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:23.294923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:23.294861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:24.173510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:24.173449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:24.950155 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:24.950107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:30.171894 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:30.171861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:39:34.096603 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.096570 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:39:34.096989 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.096796 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" podUID="c5282e8b-7550-410a-b49d-371d585a8cc5" containerName="maas-api" containerID="cri-o://0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c" gracePeriod=30 Sep 15 21:39:34.127553 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.127528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:34.224410 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.224380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:34.325348 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.325319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:34.335439 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.335418 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:39:34.425245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.425157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:34.433266 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433244 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca\") pod \"c5282e8b-7550-410a-b49d-371d585a8cc5\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " Sep 15 21:39:34.433342 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4llq\" (UniqueName: \"kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq\") pod \"c5282e8b-7550-410a-b49d-371d585a8cc5\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " Sep 15 21:39:34.433380 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433364 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs\") pod \"c5282e8b-7550-410a-b49d-371d585a8cc5\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " Sep 15 21:39:34.433440 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433427 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls\") pod \"c5282e8b-7550-410a-b49d-371d585a8cc5\" (UID: \"c5282e8b-7550-410a-b49d-371d585a8cc5\") " Sep 15 21:39:34.433709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433687 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5282e8b-7550-410a-b49d-371d585a8cc5" (UID: "c5282e8b-7550-410a-b49d-371d585a8cc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:39:34.433900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.433880 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5282e8b-7550-410a-b49d-371d585a8cc5-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.435586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.435444 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq" (OuterVolumeSpecName: "kube-api-access-l4llq") pod "c5282e8b-7550-410a-b49d-371d585a8cc5" (UID: "c5282e8b-7550-410a-b49d-371d585a8cc5"). InnerVolumeSpecName "kube-api-access-l4llq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:39:34.435992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.435957 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5282e8b-7550-410a-b49d-371d585a8cc5" (UID: "c5282e8b-7550-410a-b49d-371d585a8cc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:39:34.436105 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.436071 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5282e8b-7550-410a-b49d-371d585a8cc5" (UID: "c5282e8b-7550-410a-b49d-371d585a8cc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:39:34.534653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.534620 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.534653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.534647 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5282e8b-7550-410a-b49d-371d585a8cc5-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.534653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.534657 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4llq\" (UniqueName: \"kubernetes.io/projected/c5282e8b-7550-410a-b49d-371d585a8cc5-kube-api-access-l4llq\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:34.770069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:34.769977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:35.059725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.059627 2559 generic.go:356] "Generic (PLEG): container finished" podID="c5282e8b-7550-410a-b49d-371d585a8cc5" containerID="0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c" exitCode=0 Sep 15 21:39:35.059725 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.059678 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" Sep 15 21:39:35.059947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.059728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" event={"ID":"c5282e8b-7550-410a-b49d-371d585a8cc5","Type":"ContainerDied","Data":"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c"} Sep 15 21:39:35.059947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.059796 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552" event={"ID":"c5282e8b-7550-410a-b49d-371d585a8cc5","Type":"ContainerDied","Data":"81fa95c6d492632243caea38a038dd9b7718dd562e16a036b92d995fbe6574f7"} Sep 15 21:39:35.059947 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.059826 2559 scope.go:117] "RemoveContainer" containerID="0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c" Sep 15 21:39:35.070273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.070254 2559 scope.go:117] "RemoveContainer" containerID="0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c" Sep 15 21:39:35.070608 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:39:35.070586 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c\": container with ID starting with 0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c not found: ID does not exist" containerID="0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c" Sep 15 21:39:35.070703 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.070620 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c"} err="failed to get container status \"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c\": rpc error: code = NotFound desc = could not find container \"0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c\": container with ID starting with 0fe568537853aa78540674f6349ca58e0e40c78461e5ce2f1fdcd06fc247df7c not found: ID does not exist" Sep 15 21:39:35.083195 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.083164 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:39:35.087117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.087094 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-56820a3c-d846687ff-f5552"] Sep 15 21:39:35.567417 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:35.567380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:36.290691 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:36.290650 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5282e8b-7550-410a-b49d-371d585a8cc5" path="/var/lib/kubelet/pods/c5282e8b-7550-410a-b49d-371d585a8cc5/volumes" Sep 15 21:39:36.429266 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:36.429233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:37.131581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:37.131549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:37.342918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:37.342877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:37.963877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:37.963849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:38.757213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:38.757180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:39.476537 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:39.476502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:42.318970 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:42.318936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:39:42.371196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:42.371164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:44.488720 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:44.488687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:47.199238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:47.199163 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:39:47.227785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:47.227760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:51.148003 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:51.147974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:53.028925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:53.028893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:39:57.338072 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.338019 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" podUID="771e1482-23ea-482a-b2c2-ebe30381035a" containerName="main" containerID="cri-o://dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66" gracePeriod=45 Sep 15 21:39:57.599977 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.599922 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:39:57.648720 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.648692 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.648866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.648793 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.648866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.648831 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.649006 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.648977 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k88w5\" (UniqueName: \"kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.649070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649009 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home" (OuterVolumeSpecName: "home") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:39:57.649070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649035 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.649183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649066 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache\") pod \"771e1482-23ea-482a-b2c2-ebe30381035a\" (UID: \"771e1482-23ea-482a-b2c2-ebe30381035a\") " Sep 15 21:39:57.649314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649270 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:39:57.649511 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache" (OuterVolumeSpecName: "model-cache") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:39:57.649582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649510 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:57.649582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649531 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:57.649582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.649544 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:57.651441 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.651387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:39:57.651583 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.651489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm" (OuterVolumeSpecName: "dshm") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:39:57.651583 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.651517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5" (OuterVolumeSpecName: "kube-api-access-k88w5") pod "771e1482-23ea-482a-b2c2-ebe30381035a" (UID: "771e1482-23ea-482a-b2c2-ebe30381035a"). InnerVolumeSpecName "kube-api-access-k88w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:39:57.750916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.750888 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88w5\" (UniqueName: \"kubernetes.io/projected/771e1482-23ea-482a-b2c2-ebe30381035a-kube-api-access-k88w5\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:57.750916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.750914 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771e1482-23ea-482a-b2c2-ebe30381035a-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:57.751154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:57.750924 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771e1482-23ea-482a-b2c2-ebe30381035a-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:39:58.191095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.191059 2559 generic.go:356] "Generic (PLEG): container finished" podID="771e1482-23ea-482a-b2c2-ebe30381035a" containerID="dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66" exitCode=0 Sep 15 21:39:58.191095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.191097 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" Sep 15 21:39:58.191317 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.191144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" event={"ID":"771e1482-23ea-482a-b2c2-ebe30381035a","Type":"ContainerDied","Data":"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66"} Sep 15 21:39:58.191317 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.191184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h" event={"ID":"771e1482-23ea-482a-b2c2-ebe30381035a","Type":"ContainerDied","Data":"4e26f7fdb842e0ee6a9159279c381eea8a13b9f2dded51043cd7d23894cdfa3f"} Sep 15 21:39:58.191317 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.191203 2559 scope.go:117] "RemoveContainer" containerID="dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66" Sep 15 21:39:58.201273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.201258 2559 scope.go:117] "RemoveContainer" containerID="dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66" Sep 15 21:39:58.201643 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:39:58.201621 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66\": container with ID starting with dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66 not found: ID does not exist" containerID="dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66" Sep 15 21:39:58.201718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.201651 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66"} err="failed to get container status \"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66\": rpc error: code = NotFound desc = could not find container \"dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66\": container with ID starting with dbd4898bf1aeb9dd64e0e8124298e276ed56c57a555eab441034fb9822783d66 not found: ID does not exist" Sep 15 21:39:58.236563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.236530 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:39:58.241095 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.241065 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-56820a3c/e2e-lifecycle-model-56820a3c-kserve-5cd556fdcf-2sw7h"] Sep 15 21:39:58.291984 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:39:58.291950 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="771e1482-23ea-482a-b2c2-ebe30381035a" path="/var/lib/kubelet/pods/771e1482-23ea-482a-b2c2-ebe30381035a/volumes" Sep 15 21:40:00.023968 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.023931 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024290 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771e1482-23ea-482a-b2c2-ebe30381035a" containerName="main" Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024300 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="771e1482-23ea-482a-b2c2-ebe30381035a" containerName="main" Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024311 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5282e8b-7550-410a-b49d-371d585a8cc5" containerName="maas-api" Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024317 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5282e8b-7550-410a-b49d-371d585a8cc5" containerName="maas-api" Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024389 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="771e1482-23ea-482a-b2c2-ebe30381035a" containerName="main" Sep 15 21:40:00.024419 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.024397 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5282e8b-7550-410a-b49d-371d585a8cc5" containerName="maas-api" Sep 15 21:40:00.028928 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.028905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.031814 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.031789 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"openshift-service-ca.crt\"" Sep 15 21:40:00.032923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.032902 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"kube-root-ca.crt\"" Sep 15 21:40:00.033202 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.033178 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"default-dockercfg-xszmj\"" Sep 15 21:40:00.033271 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.033189 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"auth-test-model-38a3a3-kserve-self-signed-certs\"" Sep 15 21:40:00.039584 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.039562 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:40:00.054196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.054174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:00.074623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.074595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl58p\" (UniqueName: \"kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.074861 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.074838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.074986 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.074968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.075112 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.075003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.075112 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.075038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.075112 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.075066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.175981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl58p\" (UniqueName: \"kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176131 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176161 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176489 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176605 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.176652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.176618 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.178379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.178348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.178638 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.178622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.184644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.184622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl58p\" (UniqueName: \"kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p\") pod \"auth-test-model-38a3a3-kserve-86b946d575-xvlnn\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.338473 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.338424 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:00.613229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:00.613204 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:40:00.614790 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:40:00.614757 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49a0892_1e9b_444f_baca_f4c38bded9c6.slice/crio-a1c2b724612d541214176367f78ba462470ef4c90866fe9128c92fbf5c883675 WatchSource:0}: Error finding container a1c2b724612d541214176367f78ba462470ef4c90866fe9128c92fbf5c883675: Status 404 returned error can't find the container with id a1c2b724612d541214176367f78ba462470ef4c90866fe9128c92fbf5c883675 Sep 15 21:40:01.212287 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:01.212250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" event={"ID":"e49a0892-1e9b-444f-baca-f4c38bded9c6","Type":"ContainerStarted","Data":"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65"} Sep 15 21:40:01.212287 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:01.212294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" event={"ID":"e49a0892-1e9b-444f-baca-f4c38bded9c6","Type":"ContainerStarted","Data":"a1c2b724612d541214176367f78ba462470ef4c90866fe9128c92fbf5c883675"} Sep 15 21:40:01.232475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:01.232406 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" podStartSLOduration=1.23238769 podStartE2EDuration="1.23238769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:01.231359146 +0000 UTC m=+1449.455982153" watchObservedRunningTime="2026-09-15 21:40:01.23238769 +0000 UTC m=+1449.457010672" Sep 15 21:40:02.218370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.218331 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" podUID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" containerName="main" containerID="cri-o://3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728" gracePeriod=45 Sep 15 21:40:02.473103 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.473024 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:40:02.601182 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601149 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601193 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601214 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601251 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601325 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdch2\" (UniqueName: \"kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2\") pod \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\" (UID: \"9c5d6e48-80dc-4aaf-b059-ef1301293b99\") " Sep 15 21:40:02.601693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601530 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home" (OuterVolumeSpecName: "home") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:02.601693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601575 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:02.601693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601607 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:02.601845 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601826 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:02.601898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601851 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:02.601898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.601868 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:02.603571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.603546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm" (OuterVolumeSpecName: "dshm") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:02.603676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.603624 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2" (OuterVolumeSpecName: "kube-api-access-hdch2") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "kube-api-access-hdch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:40:02.603728 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.603683 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c5d6e48-80dc-4aaf-b059-ef1301293b99" (UID: "9c5d6e48-80dc-4aaf-b059-ef1301293b99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:40:02.703167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.703119 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdch2\" (UniqueName: \"kubernetes.io/projected/9c5d6e48-80dc-4aaf-b059-ef1301293b99-kube-api-access-hdch2\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:02.703167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.703159 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5d6e48-80dc-4aaf-b059-ef1301293b99-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:02.703167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:02.703171 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5d6e48-80dc-4aaf-b059-ef1301293b99-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:03.223533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.223495 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" containerID="3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728" exitCode=0 Sep 15 21:40:03.223533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.223534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" event={"ID":"9c5d6e48-80dc-4aaf-b059-ef1301293b99","Type":"ContainerDied","Data":"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728"} Sep 15 21:40:03.224166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.223552 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" Sep 15 21:40:03.224166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.223573 2559 scope.go:117] "RemoveContainer" containerID="3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728" Sep 15 21:40:03.224166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.223562 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7" event={"ID":"9c5d6e48-80dc-4aaf-b059-ef1301293b99","Type":"ContainerDied","Data":"aa8af8123c50676d2873da6c93060fe04ca1c55ad74059f559c9c6b584075822"} Sep 15 21:40:03.232752 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.232731 2559 scope.go:117] "RemoveContainer" containerID="3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728" Sep 15 21:40:03.233004 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:40:03.232986 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728\": container with ID starting with 3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728 not found: ID does not exist" containerID="3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728" Sep 15 21:40:03.233052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.233012 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728"} err="failed to get container status \"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728\": rpc error: code = NotFound desc = could not find container \"3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728\": container with ID starting with 3d77ae1d84c7361d7372148f2976efd71a9f705ac6758b0fb57e93d225992728 not found: ID does not exist" Sep 15 21:40:03.244935 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.244907 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:40:03.247191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:03.247168 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-b-473af5c1-kserve-65d987c4cf-5k5p7"] Sep 15 21:40:04.290765 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.290735 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" path="/var/lib/kubelet/pods/9c5d6e48-80dc-4aaf-b059-ef1301293b99/volumes" Sep 15 21:40:04.736100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.736059 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:40:04.736447 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.736436 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" containerName="main" Sep 15 21:40:04.736514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.736449 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" containerName="main" Sep 15 21:40:04.736559 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.736550 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5d6e48-80dc-4aaf-b059-ef1301293b99" containerName="main" Sep 15 21:40:04.740806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.740787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.744588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.744568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"distinct-w4-ac4411-kserve-self-signed-certs\"" Sep 15 21:40:04.756698 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.756679 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:40:04.763267 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.763237 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:04.823766 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.823900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.823900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.823900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.824001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.824001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.823957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p249k\" (UniqueName: \"kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.924792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.924792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925020 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.924952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p249k\" (UniqueName: \"kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.925192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925290 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.925245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.925344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.925296 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.927015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.926990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.927332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.927307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:04.937176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:04.937148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p249k\" (UniqueName: \"kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k\") pod \"distinct-w4-ac4411-kserve-5fb4777945-2v57j\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:05.050521 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:05.050429 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:05.333380 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:40:05.333344 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf802ab4b_7b3a_47b2_8380_65a77dd4bf87.slice/crio-3eb6dfeb6e3a2c5984ab9ce3b10cea5f77ba4cf527d24443f81ff44573e60a18 WatchSource:0}: Error finding container 3eb6dfeb6e3a2c5984ab9ce3b10cea5f77ba4cf527d24443f81ff44573e60a18: Status 404 returned error can't find the container with id 3eb6dfeb6e3a2c5984ab9ce3b10cea5f77ba4cf527d24443f81ff44573e60a18 Sep 15 21:40:05.334032 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:05.334007 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:40:06.240920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:06.240881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" event={"ID":"f802ab4b-7b3a-47b2-8380-65a77dd4bf87","Type":"ContainerStarted","Data":"84befaf220146a9c7830546efa7bb9d4c63e869f3a02db3a583b4cb545b13d21"} Sep 15 21:40:06.241089 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:06.240933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" event={"ID":"f802ab4b-7b3a-47b2-8380-65a77dd4bf87","Type":"ContainerStarted","Data":"3eb6dfeb6e3a2c5984ab9ce3b10cea5f77ba4cf527d24443f81ff44573e60a18"} Sep 15 21:40:06.258773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:06.258716 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" podStartSLOduration=2.258696509 podStartE2EDuration="2.258696509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:06.257343738 +0000 UTC m=+1454.481966746" watchObservedRunningTime="2026-09-15 21:40:06.258696509 +0000 UTC m=+1454.483319514" Sep 15 21:40:09.022088 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:09.022051 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:40:10.338988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:10.338958 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:10.338988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:10.338997 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:10.351713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:10.351687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:11.279086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:11.279059 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:40:15.050636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:15.050600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:15.051042 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:15.050652 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:15.063433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:15.063404 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:15.301683 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:15.301595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:40:18.609395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:18.609341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:19.518406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:19.518368 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:22.069829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:22.069787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:22.609433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:22.609369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:24.077067 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:24.077036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:25.177898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.177854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:25.254587 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.254532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:25.409938 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.409902 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:40:25.410226 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.410169 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" containerID="cri-o://3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c" gracePeriod=30 Sep 15 21:40:25.437086 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.437017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:25.636355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.636314 2559 patch_prober.go:28] interesting pod/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:40:25.636531 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:25.636387 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:40:26.641778 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:26.641744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:26.736066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:26.736038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:26.836829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:26.836803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:27.397355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:27.397326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:27.496221 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:27.496189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:27.600670 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:27.600643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:28.884331 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:28.884301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:30.660549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.660525 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.664852 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.664923 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665031 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665103 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665128 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665168 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665337 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:40:30.665982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.665936 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:30.666701 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666124 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:30.666701 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666181 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.666701 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666290 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fmls\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls\") pod \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\" (UID: \"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb\") " Sep 15 21:40:30.667510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666881 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.667510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666905 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.667510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666922 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.667510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666942 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.667510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.666955 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.668693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.667942 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:40:30.674730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.669532 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls" (OuterVolumeSpecName: "kube-api-access-9fmls") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "kube-api-access-9fmls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:40:30.674730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.669873 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:40:30.675548 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.675524 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" (UID: "9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:40:30.768066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.768031 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.768066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.768061 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.768066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.768071 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fmls\" (UniqueName: \"kubernetes.io/projected/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-kube-api-access-9fmls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:30.768289 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:30.768082 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:40:31.372443 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.372410 2559 generic.go:356] "Generic (PLEG): container finished" podID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerID="3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c" exitCode=0 Sep 15 21:40:31.372656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.372473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerDied","Data":"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c"} Sep 15 21:40:31.372656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.372505 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" Sep 15 21:40:31.372656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.372546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk" event={"ID":"9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb","Type":"ContainerDied","Data":"fd3c79b6a07bd21242ef37edfea5f41fe71b67b34da7b9061736ff25ac102f72"} Sep 15 21:40:31.372656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.372579 2559 scope.go:117] "RemoveContainer" containerID="3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c" Sep 15 21:40:31.382025 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.382001 2559 scope.go:117] "RemoveContainer" containerID="27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91" Sep 15 21:40:31.390300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.390282 2559 scope.go:117] "RemoveContainer" containerID="3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c" Sep 15 21:40:31.390580 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:40:31.390561 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c\": container with ID starting with 3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c not found: ID does not exist" containerID="3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c" Sep 15 21:40:31.390646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.390588 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c"} err="failed to get container status \"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c\": rpc error: code = NotFound desc = could not find container \"3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c\": container with ID starting with 3d00524702e7bacb20f62accbaee3f956efc06ef0a725db3a64a8bd5997c035c not found: ID does not exist" Sep 15 21:40:31.390646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.390604 2559 scope.go:117] "RemoveContainer" containerID="27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91" Sep 15 21:40:31.390852 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:40:31.390834 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91\": container with ID starting with 27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91 not found: ID does not exist" containerID="27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91" Sep 15 21:40:31.390905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.390862 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91"} err="failed to get container status \"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91\": rpc error: code = NotFound desc = could not find container \"27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91\": container with ID starting with 27c3394498b0b7f6357cf2961b5d43829cd1c8ddbde1fe743506666b232c9a91 not found: ID does not exist" Sep 15 21:40:31.395617 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.395597 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:40:31.399049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:31.399029 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-56820a3c-openshift-default-67484d975b-9bnmk"] Sep 15 21:40:32.290338 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:32.290305 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" path="/var/lib/kubelet/pods/9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb/volumes" Sep 15 21:40:34.752228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:34.752192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:34.853621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:34.853592 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:35.396216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:35.396188 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:41.087905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:41.087871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:41.196558 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:41.196525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:41.294392 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:41.294360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:44.417137 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417102 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:40:44.417534 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417510 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.417534 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417523 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.417534 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417532 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.417653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417538 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.417653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417596 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.417653 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.417603 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd73aed-e40b-4d5a-8cfa-91f39b8c43cb" containerName="istio-proxy" Sep 15 21:40:44.422138 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.422120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.424568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.424547 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"distinct-2-w4-ac4411-kserve-self-signed-certs\"" Sep 15 21:40:44.428711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.428686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:40:44.487885 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.487843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.487885 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.487917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.488161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.487951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.488161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.487979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g76zb\" (UniqueName: \"kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.488161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.488076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.488161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.488142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.520165 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.520135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:44.589596 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.589756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.589756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.589756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g76zb\" (UniqueName: \"kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.589756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.589975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.589779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.590113 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.590088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.590162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.590125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.590162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.590151 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.591937 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.591909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.592213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.592194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.603245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.603218 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g76zb\" (UniqueName: \"kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb\") pod \"distinct-2-w4-ac4411-kserve-544c89554b-7j4j6\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:44.648162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.648134 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:40:44.732131 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:44.732064 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:45.010829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:45.010800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:40:45.012563 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:40:45.012542 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd51cd259_fa1a_4258_a90c_17249735fc06.slice/crio-3f8c4283bf1d21140b1c941885cfcd286baed90db8f22d270f8daf616001e8d5 WatchSource:0}: Error finding container 3f8c4283bf1d21140b1c941885cfcd286baed90db8f22d270f8daf616001e8d5: Status 404 returned error can't find the container with id 3f8c4283bf1d21140b1c941885cfcd286baed90db8f22d270f8daf616001e8d5 Sep 15 21:40:45.447866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:45.447832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" event={"ID":"d51cd259-fa1a-4258-a90c-17249735fc06","Type":"ContainerStarted","Data":"414b5144760eda56a881d147a7e95d1bd3cbd0c3df878ae8b81c9947601cd940"} Sep 15 21:40:45.448240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:45.447882 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" event={"ID":"d51cd259-fa1a-4258-a90c-17249735fc06","Type":"ContainerStarted","Data":"3f8c4283bf1d21140b1c941885cfcd286baed90db8f22d270f8daf616001e8d5"} Sep 15 21:40:45.467305 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:45.467243 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" podStartSLOduration=1.467224137 podStartE2EDuration="1.467224137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:45.465182036 +0000 UTC m=+1493.689805040" watchObservedRunningTime="2026-09-15 21:40:45.467224137 +0000 UTC m=+1493.691847142" Sep 15 21:40:51.745304 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:51.745272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:51.970230 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:51.970195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:40:54.732777 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:54.732736 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:54.733126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:54.732820 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:54.745358 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:54.745338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:55.512079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:55.512048 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:40:56.451429 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:40:56.451396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:12.853961 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:12.853925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:15.525491 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:15.525393 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:41:19.034759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:19.034719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:19.134797 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:19.134772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:19.387030 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:19.387000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:19.701258 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:19.701168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:19.835313 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:19.835280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:21.672356 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:21.672320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:22.356913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:22.356879 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:22.446863 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:22.446831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:22.528688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:22.528661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:23.491101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:23.491068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:24.283900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:24.283859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:25.162156 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:25.162121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:25.948956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:25.948922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:26.659917 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:26.659882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:27.436222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.436187 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:41:27.440027 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.440003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.442757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.442732 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b0aa2e-serving-cert\"" Sep 15 21:41:27.450710 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.450686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:41:27.472850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.472828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:27.495126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.495098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psslk\" (UniqueName: \"kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.495222 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.495144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.495264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.495223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.495310 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.495285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.570586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.570555 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:27.596718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.596694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psslk\" (UniqueName: \"kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.596836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.596728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.596836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.596760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.596836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.596810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.596967 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:41:27.596929 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-serving-cert: secret "maas-api-e2e-api-a-b0aa2e-serving-cert" not found Sep 15 21:41:27.597012 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:41:27.597001 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls podName:4440b02a-d89e-4d99-8116-54e72177b6ce nodeName:}" failed. No retries permitted until 2026-09-15 21:41:28.096981076 +0000 UTC m=+1536.321604063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls") pod "maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" (UID: "4440b02a-d89e-4d99-8116-54e72177b6ce") : secret "maas-api-e2e-api-a-b0aa2e-serving-cert" not found Sep 15 21:41:27.597450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.597432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.599430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.599404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.605718 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.605700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psslk\" (UniqueName: \"kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:27.811510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.811416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:27.952300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:27.952268 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:28.052281 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.052252 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:28.102432 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.102401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:28.104696 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.104675 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") pod \"maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:28.352711 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.352606 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:28.627567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.627539 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:41:28.629089 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:41:28.629064 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4440b02a_d89e_4d99_8116_54e72177b6ce.slice/crio-0abf704122ee7b45031fe45ed4f87aa7817e6a27c62cf6bac2b0847c1a086e89 WatchSource:0}: Error finding container 0abf704122ee7b45031fe45ed4f87aa7817e6a27c62cf6bac2b0847c1a086e89: Status 404 returned error can't find the container with id 0abf704122ee7b45031fe45ed4f87aa7817e6a27c62cf6bac2b0847c1a086e89 Sep 15 21:41:28.630469 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.630441 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:41:28.676390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:28.676364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" event={"ID":"4440b02a-d89e-4d99-8116-54e72177b6ce","Type":"ContainerStarted","Data":"0abf704122ee7b45031fe45ed4f87aa7817e6a27c62cf6bac2b0847c1a086e89"} Sep 15 21:41:29.457850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:29.457822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:29.682590 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:29.682511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" event={"ID":"4440b02a-d89e-4d99-8116-54e72177b6ce","Type":"ContainerStarted","Data":"b28e752c98e0760adb022f6f9cf94e92a00d32e6d1a63890d4b2fc18450131ea"} Sep 15 21:41:29.682776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:29.682640 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:29.701828 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:29.701785 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" podStartSLOduration=1.940663525 podStartE2EDuration="2.701770365s" totalImagesPullingTime="761.10684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:29.699757765 +0000 UTC m=+1537.924380774" watchObservedRunningTime="2026-09-15 21:41:29.701770365 +0000 UTC m=+1537.926393369" Sep 15 21:41:35.694049 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:35.694017 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:41:46.448007 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:46.447968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:49.666591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:49.666559 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:50.452206 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:50.452175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:51.157200 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:51.157161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:51.918395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:51.918364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:52.388294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:52.388266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:52.865571 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:52.865539 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:52.960447 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:52.960416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:53.148445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:53.148359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:53.649050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:53.649015 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:53.720632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:53.720557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:54.453430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:54.453398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:55.196692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:55.196665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:55.916916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:55.916883 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:56.373354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:56.373320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:56.469214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:56.469180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:56.992255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:56.992221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:57.833190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:57.833160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:58.693164 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:58.693118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:41:59.389702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:41:59.389667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:00.171886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:00.171853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:00.889028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:00.888997 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:01.638295 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:01.638262 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:02.028129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.028047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:42:02.454735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.454704 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:42:02.458439 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.458416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.461872 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.461847 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-57a27b-serving-cert\"" Sep 15 21:42:02.473375 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.473349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:42:02.493550 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.493526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:02.517923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.517894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kz9\" (UniqueName: \"kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.518034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.517959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.518034 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.518018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.518115 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.518062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.592769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.592711 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:02.618903 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.618881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.619032 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.618910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.619032 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.619001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kz9\" (UniqueName: \"kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.619133 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:42:02.619030 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-serving-cert: secret "maas-api-e2e-api-b-57a27b-serving-cert" not found Sep 15 21:42:02.619133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.619054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.619133 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:42:02.619103 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls podName:a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9 nodeName:}" failed. No retries permitted until 2026-09-15 21:42:03.119082678 +0000 UTC m=+1571.343705680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls") pod "maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" (UID: "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9") : secret "maas-api-e2e-api-b-57a27b-serving-cert" not found Sep 15 21:42:02.619772 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.619753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.621393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.621378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.628654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.628631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kz9\" (UniqueName: \"kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:02.816210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:02.816120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:03.123815 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.123774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:03.126225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.126203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") pod \"maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:03.135512 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.135468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:03.228289 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.228254 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:03.370389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.370358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:03.643210 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.643175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:42:03.646111 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:42:03.646082 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eb3bf1_9c17_414b_9aaf_619ba17c7dc9.slice/crio-a0fc5b1a1834f9711d935a0055f2dadf70744258ab46b16090fa5e937964fe9e WatchSource:0}: Error finding container a0fc5b1a1834f9711d935a0055f2dadf70744258ab46b16090fa5e937964fe9e: Status 404 returned error can't find the container with id a0fc5b1a1834f9711d935a0055f2dadf70744258ab46b16090fa5e937964fe9e Sep 15 21:42:03.862642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:03.862598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" event={"ID":"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9","Type":"ContainerStarted","Data":"a0fc5b1a1834f9711d935a0055f2dadf70744258ab46b16090fa5e937964fe9e"} Sep 15 21:42:04.870543 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:04.870508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" event={"ID":"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9","Type":"ContainerStarted","Data":"5c2024346680939347f29d58b12cb0770f8a2e643b2378a93e6282d4f5960fc5"} Sep 15 21:42:04.870910 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:04.870709 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:04.889131 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:04.889087 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" podStartSLOduration=2.398626114 podStartE2EDuration="2.88907477s" totalImagesPullingTime="490.448656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:04.887632148 +0000 UTC m=+1573.112255152" watchObservedRunningTime="2026-09-15 21:42:04.88907477 +0000 UTC m=+1573.113697773" Sep 15 21:42:08.004282 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:08.004253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:10.880174 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:10.880147 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:42:23.673838 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:23.673805 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:25.062916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:25.062882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:25.844764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:25.844725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:26.327647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:26.327614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:26.575207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:26.575171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:27.524237 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:27.524206 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:28.044212 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:28.044180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:28.302257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:28.302161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:29.447382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:29.447351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:30.181038 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:30.181007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:31.187596 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:31.187554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:32.217923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:32.217893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:33.045652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:33.045612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:34.029572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:34.029541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:34.930951 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:34.930916 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:35.793826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:35.793794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:36.595577 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:36.595545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:37.389761 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:37.389729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:39.359050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:39.359020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:43.068506 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:43.068448 2559 generic.go:356] "Generic (PLEG): container finished" podID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerID="2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314" exitCode=137 Sep 15 21:42:43.068985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:43.068522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerDied","Data":"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314"} Sep 15 21:42:43.068985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:43.068805 2559 scope.go:117] "RemoveContainer" containerID="2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314" Sep 15 21:42:44.077244 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:44.077205 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerStarted","Data":"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2"} Sep 15 21:42:44.077649 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:44.077426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:42:45.206186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:45.206101 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:45.927742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:45.927710 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:42:45.928181 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:45.928151 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" podUID="4440b02a-d89e-4d99-8116-54e72177b6ce" containerName="maas-api" containerID="cri-o://b28e752c98e0760adb022f6f9cf94e92a00d32e6d1a63890d4b2fc18450131ea" gracePeriod=30 Sep 15 21:42:45.959417 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:45.959391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.062550 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.062437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.089664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.089632 2559 generic.go:356] "Generic (PLEG): container finished" podID="4440b02a-d89e-4d99-8116-54e72177b6ce" containerID="b28e752c98e0760adb022f6f9cf94e92a00d32e6d1a63890d4b2fc18450131ea" exitCode=0 Sep 15 21:42:46.089664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.089667 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" event={"ID":"4440b02a-d89e-4d99-8116-54e72177b6ce","Type":"ContainerDied","Data":"b28e752c98e0760adb022f6f9cf94e92a00d32e6d1a63890d4b2fc18450131ea"} Sep 15 21:42:46.170357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.170332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.187696 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.187647 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:42:46.227216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psslk\" (UniqueName: \"kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk\") pod \"4440b02a-d89e-4d99-8116-54e72177b6ce\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " Sep 15 21:42:46.227676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca\") pod \"4440b02a-d89e-4d99-8116-54e72177b6ce\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " Sep 15 21:42:46.227676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227267 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs\") pod \"4440b02a-d89e-4d99-8116-54e72177b6ce\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " Sep 15 21:42:46.227676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227381 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") pod \"4440b02a-d89e-4d99-8116-54e72177b6ce\" (UID: \"4440b02a-d89e-4d99-8116-54e72177b6ce\") " Sep 15 21:42:46.227676 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227664 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4440b02a-d89e-4d99-8116-54e72177b6ce" (UID: "4440b02a-d89e-4d99-8116-54e72177b6ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:42:46.227903 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.227809 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4440b02a-d89e-4d99-8116-54e72177b6ce-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:42:46.229574 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.229538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk" (OuterVolumeSpecName: "kube-api-access-psslk") pod "4440b02a-d89e-4d99-8116-54e72177b6ce" (UID: "4440b02a-d89e-4d99-8116-54e72177b6ce"). InnerVolumeSpecName "kube-api-access-psslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:42:46.229705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.229637 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4440b02a-d89e-4d99-8116-54e72177b6ce" (UID: "4440b02a-d89e-4d99-8116-54e72177b6ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:46.229877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.229847 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4440b02a-d89e-4d99-8116-54e72177b6ce" (UID: "4440b02a-d89e-4d99-8116-54e72177b6ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:42:46.251589 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.251566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.328519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.328490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.328751 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.328722 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psslk\" (UniqueName: \"kubernetes.io/projected/4440b02a-d89e-4d99-8116-54e72177b6ce-kube-api-access-psslk\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:42:46.329566 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.328756 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:42:46.329566 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.328773 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4440b02a-d89e-4d99-8116-54e72177b6ce-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:42:46.598705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.598671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:46.769850 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:46.769800 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:47.096225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.096196 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" Sep 15 21:42:47.096225 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.096216 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd" event={"ID":"4440b02a-d89e-4d99-8116-54e72177b6ce","Type":"ContainerDied","Data":"0abf704122ee7b45031fe45ed4f87aa7817e6a27c62cf6bac2b0847c1a086e89"} Sep 15 21:42:47.096430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.096278 2559 scope.go:117] "RemoveContainer" containerID="b28e752c98e0760adb022f6f9cf94e92a00d32e6d1a63890d4b2fc18450131ea" Sep 15 21:42:47.150000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.149971 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:42:47.157257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.157236 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b0aa2e-7c8df7b657-9mwpd"] Sep 15 21:42:47.869146 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:47.869117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:48.290640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:48.290570 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4440b02a-d89e-4d99-8116-54e72177b6ce" path="/var/lib/kubelet/pods/4440b02a-d89e-4d99-8116-54e72177b6ce/volumes" Sep 15 21:42:48.629145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:48.629109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:49.491026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:49.490998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:50.228250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:50.228221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:50.972411 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:50.972382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:51.929703 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:51.929648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:52.575159 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:52.575130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:52.774196 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:52.774114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:52.872914 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:52.872878 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:53.132070 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:53.132033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:53.643283 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:53.643249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:54.463885 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:54.463856 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:55.084435 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:55.084404 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:42:55.297996 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:55.297957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:56.194726 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:56.194690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:57.164775 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:57.164741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:57.880133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:57.880089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:58.641145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:58.641114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:42:59.475073 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:42:59.475046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:00.159549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:00.159519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:08.991328 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:08.991296 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:43:08.991791 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:08.991549 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" podUID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" containerName="maas-api" containerID="cri-o://5c2024346680939347f29d58b12cb0770f8a2e643b2378a93e6282d4f5960fc5" gracePeriod=30 Sep 15 21:43:09.009255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.009231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.085278 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.085227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.146051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.146022 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.213147 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.213116 2559 generic.go:356] "Generic (PLEG): container finished" podID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" containerID="5c2024346680939347f29d58b12cb0770f8a2e643b2378a93e6282d4f5960fc5" exitCode=0 Sep 15 21:43:09.213312 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.213185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" event={"ID":"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9","Type":"ContainerDied","Data":"5c2024346680939347f29d58b12cb0770f8a2e643b2378a93e6282d4f5960fc5"} Sep 15 21:43:09.218735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.218715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.237015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.236993 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:43:09.293719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.293655 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.358538 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.358509 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca\") pod \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " Sep 15 21:43:09.358668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.358558 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs\") pod \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " Sep 15 21:43:09.358668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.358609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4kz9\" (UniqueName: \"kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9\") pod \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " Sep 15 21:43:09.358668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.358650 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") pod \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\" (UID: \"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9\") " Sep 15 21:43:09.359282 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.358991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" (UID: "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:09.361633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.361605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" (UID: "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:09.361730 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.361695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" (UID: "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:09.362602 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.362577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9" (OuterVolumeSpecName: "kube-api-access-n4kz9") pod "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" (UID: "a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9"). InnerVolumeSpecName "kube-api-access-n4kz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:09.460355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.460322 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.460355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.460357 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.460618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.460371 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kz9\" (UniqueName: \"kubernetes.io/projected/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-kube-api-access-n4kz9\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.460618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.460385 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:09.590038 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.590005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:09.686108 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:09.686075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:10.219585 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.219557 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" Sep 15 21:43:10.219585 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.219567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5" event={"ID":"a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9","Type":"ContainerDied","Data":"a0fc5b1a1834f9711d935a0055f2dadf70744258ab46b16090fa5e937964fe9e"} Sep 15 21:43:10.219992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.219644 2559 scope.go:117] "RemoveContainer" containerID="5c2024346680939347f29d58b12cb0770f8a2e643b2378a93e6282d4f5960fc5" Sep 15 21:43:10.244842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.244814 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:43:10.248733 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.248713 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57a27b-66fd7d7757-92zw5"] Sep 15 21:43:10.290360 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.290325 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" path="/var/lib/kubelet/pods/a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9/volumes" Sep 15 21:43:10.368771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:10.368744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:11.113784 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:11.113754 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:11.920667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:11.920634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:12.662524 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:12.662492 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:13.430097 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:13.430061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:43:13.495000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:13.494965 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:14.193259 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:14.193219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:15.819116 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:15.819088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:15.889800 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:15.889770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:16.105081 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:16.105055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:17.433245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:17.433214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:18.369079 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:18.369045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:19.090858 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:19.090828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:19.965995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:19.965950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:20.659888 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:20.659858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:21.454738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:21.454702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:22.204720 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:22.204690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:26.928151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:26.928113 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:43:26.928613 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:26.928388 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" podUID="646e5395-813e-4dd1-8874-0100952e1d08" containerName="maas-api" containerID="cri-o://70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10" gracePeriod=30 Sep 15 21:43:26.951642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:26.951612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:27.051239 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.051210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:27.149141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.149106 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:27.184820 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.184756 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:43:27.248341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.248310 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sck2h\" (UniqueName: \"kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h\") pod \"646e5395-813e-4dd1-8874-0100952e1d08\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " Sep 15 21:43:27.248341 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.248340 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs\") pod \"646e5395-813e-4dd1-8874-0100952e1d08\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " Sep 15 21:43:27.248594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.248375 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls\") pod \"646e5395-813e-4dd1-8874-0100952e1d08\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " Sep 15 21:43:27.248594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.248398 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca\") pod \"646e5395-813e-4dd1-8874-0100952e1d08\" (UID: \"646e5395-813e-4dd1-8874-0100952e1d08\") " Sep 15 21:43:27.248905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.248879 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "646e5395-813e-4dd1-8874-0100952e1d08" (UID: "646e5395-813e-4dd1-8874-0100952e1d08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:27.249395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.249369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:27.250673 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.250646 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "646e5395-813e-4dd1-8874-0100952e1d08" (UID: "646e5395-813e-4dd1-8874-0100952e1d08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:27.250763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.250668 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "646e5395-813e-4dd1-8874-0100952e1d08" (UID: "646e5395-813e-4dd1-8874-0100952e1d08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:27.250763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.250681 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h" (OuterVolumeSpecName: "kube-api-access-sck2h") pod "646e5395-813e-4dd1-8874-0100952e1d08" (UID: "646e5395-813e-4dd1-8874-0100952e1d08"). InnerVolumeSpecName "kube-api-access-sck2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:27.311826 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.311790 2559 generic.go:356] "Generic (PLEG): container finished" podID="646e5395-813e-4dd1-8874-0100952e1d08" containerID="70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10" exitCode=0 Sep 15 21:43:27.311990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.311846 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" Sep 15 21:43:27.311990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.311867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" event={"ID":"646e5395-813e-4dd1-8874-0100952e1d08","Type":"ContainerDied","Data":"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10"} Sep 15 21:43:27.311990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.311921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz" event={"ID":"646e5395-813e-4dd1-8874-0100952e1d08","Type":"ContainerDied","Data":"534ed8e947fd426874810a77756eb958836d223cbc64cd80cd797f19c4ccef45"} Sep 15 21:43:27.311990 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.311943 2559 scope.go:117] "RemoveContainer" containerID="70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10" Sep 15 21:43:27.321227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.321211 2559 scope.go:117] "RemoveContainer" containerID="70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10" Sep 15 21:43:27.321528 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:43:27.321507 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10\": container with ID starting with 70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10 not found: ID does not exist" containerID="70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10" Sep 15 21:43:27.321600 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.321536 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10"} err="failed to get container status \"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10\": rpc error: code = NotFound desc = could not find container \"70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10\": container with ID starting with 70a070c49011bb6704b3975f1661ec6e9bda09f1ee8dd287751d152a9f9ceb10 not found: ID does not exist" Sep 15 21:43:27.337549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.337517 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:43:27.341654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.341632 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2d4fab-7bcd9d9b5d-mbskz"] Sep 15 21:43:27.349959 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.349940 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sck2h\" (UniqueName: \"kubernetes.io/projected/646e5395-813e-4dd1-8874-0100952e1d08-kube-api-access-sck2h\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.350028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.349962 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.350028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.349973 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646e5395-813e-4dd1-8874-0100952e1d08-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.350028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.349981 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e5395-813e-4dd1-8874-0100952e1d08-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:27.561184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.561096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:27.661929 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:27.661889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:28.290368 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:28.290336 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646e5395-813e-4dd1-8874-0100952e1d08" path="/var/lib/kubelet/pods/646e5395-813e-4dd1-8874-0100952e1d08/volumes" Sep 15 21:43:28.470845 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:28.470812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:29.230848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:29.230812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:30.040256 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:30.040227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:30.952916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:30.952883 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:31.870191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:31.870156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:33.483922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:33.483889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:33.586794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:33.586762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:33.740476 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:33.740371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:34.429977 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:34.429946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:35.211631 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:35.211543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:36.063942 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:36.063905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:36.515995 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:36.515955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:37.062687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.062652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:37.223763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.223734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:37.333290 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333254 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs"] Sep 15 21:43:37.333705 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333689 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4440b02a-d89e-4d99-8116-54e72177b6ce" containerName="maas-api" Sep 15 21:43:37.333754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333707 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4440b02a-d89e-4d99-8116-54e72177b6ce" containerName="maas-api" Sep 15 21:43:37.333754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333724 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646e5395-813e-4dd1-8874-0100952e1d08" containerName="maas-api" Sep 15 21:43:37.333754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333732 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="646e5395-813e-4dd1-8874-0100952e1d08" containerName="maas-api" Sep 15 21:43:37.333844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333755 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" containerName="maas-api" Sep 15 21:43:37.333844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333761 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" containerName="maas-api" Sep 15 21:43:37.333844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333828 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="646e5395-813e-4dd1-8874-0100952e1d08" containerName="maas-api" Sep 15 21:43:37.333844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333836 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1eb3bf1-9c17-414b-9aaf-619ba17c7dc9" containerName="maas-api" Sep 15 21:43:37.333844 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.333845 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4440b02a-d89e-4d99-8116-54e72177b6ce" containerName="maas-api" Sep 15 21:43:37.336706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.336690 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:37.339484 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.339442 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\"" Sep 15 21:43:37.346543 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.346524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs"] Sep 15 21:43:37.453856 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.453824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs\" (UID: \"263fd6e8-fb5b-4be5-b840-b8c204e63ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:37.555035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.555006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs\" (UID: \"263fd6e8-fb5b-4be5-b840-b8c204e63ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:37.555646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.555625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs\" (UID: \"263fd6e8-fb5b-4be5-b840-b8c204e63ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:37.649008 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.648926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:37.922714 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.922687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs"] Sep 15 21:43:37.924923 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:43:37.924891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263fd6e8_fb5b_4be5_b840_b8c204e63ce3.slice/crio-b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f WatchSource:0}: Error finding container b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f: Status 404 returned error can't find the container with id b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f Sep 15 21:43:37.938344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:37.938324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:38.371384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:38.371350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" event={"ID":"263fd6e8-fb5b-4be5-b840-b8c204e63ce3","Type":"ContainerStarted","Data":"960a993697b1dd17ecb8f774e3fc843b502b840f0f7f1e4f4d87115336cf78e5"} Sep 15 21:43:38.371567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:38.371411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" event={"ID":"263fd6e8-fb5b-4be5-b840-b8c204e63ce3","Type":"ContainerStarted","Data":"b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f"} Sep 15 21:43:38.386256 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:38.386209 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" podStartSLOduration=1.386194196 podStartE2EDuration="1.386194196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:38.385920764 +0000 UTC m=+1666.610543782" watchObservedRunningTime="2026-09-15 21:43:38.386194196 +0000 UTC m=+1666.610817202" Sep 15 21:43:38.667355 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:38.667322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:39.378519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:39.378482 2559 generic.go:356] "Generic (PLEG): container finished" podID="263fd6e8-fb5b-4be5-b840-b8c204e63ce3" containerID="960a993697b1dd17ecb8f774e3fc843b502b840f0f7f1e4f4d87115336cf78e5" exitCode=0 Sep 15 21:43:39.378704 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:39.378649 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" event={"ID":"263fd6e8-fb5b-4be5-b840-b8c204e63ce3","Type":"ContainerDied","Data":"960a993697b1dd17ecb8f774e3fc843b502b840f0f7f1e4f4d87115336cf78e5"} Sep 15 21:43:39.472923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:39.472893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:40.219035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.218985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:40.511379 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.511353 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:40.684438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.684404 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca\") pod \"263fd6e8-fb5b-4be5-b840-b8c204e63ce3\" (UID: \"263fd6e8-fb5b-4be5-b840-b8c204e63ce3\") " Sep 15 21:43:40.684752 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.684728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "263fd6e8-fb5b-4be5-b840-b8c204e63ce3" (UID: "263fd6e8-fb5b-4be5-b840-b8c204e63ce3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:43:40.785757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.785683 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/263fd6e8-fb5b-4be5-b840-b8c204e63ce3-maas-api-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:40.917582 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:40.917554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:41.389769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.389742 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" Sep 15 21:43:41.390129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.389774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs" event={"ID":"263fd6e8-fb5b-4be5-b840-b8c204e63ce3","Type":"ContainerDied","Data":"b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f"} Sep 15 21:43:41.390129 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.389813 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b499b7d7f73aa366ded0e99e83ff3f3cf1f08302d170c6bda81051100ef89c0f" Sep 15 21:43:41.542401 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.542370 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:43:41.606975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.606943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:41.884228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:41.884194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:42.206244 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:42.206165 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:43:42.211686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:42.211653 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:43:42.220922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:42.220883 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:43:42.233413 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:42.233390 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:43:42.294065 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:42.294032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:44.234308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:44.234279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:44.966293 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:44.966257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:45.041028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:45.040988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:45.140768 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:45.140736 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:45.206628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:45.206599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:45.535817 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:45.535781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:45.631643 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:45.631614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:46.367960 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:46.367925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:47.175708 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:47.175673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:48.055724 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:48.055696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:49.927792 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:49.927758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:43:51.551161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:51.551120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:51.618243 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:51.618214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:51.831091 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:51.831050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.047989 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.047950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.149375 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.149291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.262285 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.262245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.360450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.360418 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.713636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.713602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.727642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.727614 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5"] Sep 15 21:43:52.735555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.735531 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ac4411-29825130-qbmb5"] Sep 15 21:43:52.825540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.825406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:52.931397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:52.931357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:53.023549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.023504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:53.024285 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.024264 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs"] Sep 15 21:43:53.029641 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.029617 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ac4411-ht6bs"] Sep 15 21:43:53.124470 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.124344 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:53.378257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.378163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:53.940248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:53.940216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:54.290254 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:54.290164 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e380ae3-ba23-4458-a3a2-1300ee4e1dff" path="/var/lib/kubelet/pods/1e380ae3-ba23-4458-a3a2-1300ee4e1dff/volumes" Sep 15 21:43:54.290633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:54.290612 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263fd6e8-fb5b-4be5-b840-b8c204e63ce3" path="/var/lib/kubelet/pods/263fd6e8-fb5b-4be5-b840-b8c204e63ce3/volumes" Sep 15 21:43:54.659404 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:54.659356 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:55.458238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:55.458195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:56.562365 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.562301 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" podUID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" containerName="main" containerID="cri-o://b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c" gracePeriod=45 Sep 15 21:43:56.826416 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.826395 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:43:56.925922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.925889 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.925939 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926014 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926072 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926100 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcc58\" (UniqueName: \"kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926210 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs\") pod \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\" (UID: \"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246\") " Sep 15 21:43:56.926405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.926405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926373 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home" (OuterVolumeSpecName: "home") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.926533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.926533 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926520 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.926623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.926540 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:56.928253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.928221 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm" (OuterVolumeSpecName: "dshm") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:56.928370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.928353 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:56.928414 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:56.928363 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58" (OuterVolumeSpecName: "kube-api-access-qcc58") pod "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" (UID: "5e29fee9-f4c5-49b9-a2ed-9a2c398a5246"). InnerVolumeSpecName "kube-api-access-qcc58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.027052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.027020 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.027052 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.027048 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.027227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.027058 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcc58\" (UniqueName: \"kubernetes.io/projected/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-kube-api-access-qcc58\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.027227 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.027069 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.226162 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.226075 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" podUID="d51cd259-fa1a-4258-a90c-17249735fc06" containerName="main" containerID="cri-o://414b5144760eda56a881d147a7e95d1bd3cbd0c3df878ae8b81c9947601cd940" gracePeriod=45 Sep 15 21:43:57.230816 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.230754 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" podUID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" containerName="main" containerID="cri-o://84befaf220146a9c7830546efa7bb9d4c63e869f3a02db3a583b4cb545b13d21" gracePeriod=45 Sep 15 21:43:57.239693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.239656 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" podUID="950d6cb1-65c7-4a68-854e-79eb133a6e25" containerName="main" containerID="cri-o://8d92ff5fd39ed4e9638377041ff4b73c8fb2d2c3ca9e05469644dfd64d4a29b5" gracePeriod=45 Sep 15 21:43:57.252395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.252367 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" podUID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" containerName="main" containerID="cri-o://99733f38d469198f0ede59ffbd18a7134b5317a99ca064ab9940c44b2b703f47" gracePeriod=45 Sep 15 21:43:57.476395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.476309 2559 generic.go:356] "Generic (PLEG): container finished" podID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" containerID="99733f38d469198f0ede59ffbd18a7134b5317a99ca064ab9940c44b2b703f47" exitCode=0 Sep 15 21:43:57.476575 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.476389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" event={"ID":"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5","Type":"ContainerDied","Data":"99733f38d469198f0ede59ffbd18a7134b5317a99ca064ab9940c44b2b703f47"} Sep 15 21:43:57.478769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.478746 2559 generic.go:356] "Generic (PLEG): container finished" podID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" containerID="84befaf220146a9c7830546efa7bb9d4c63e869f3a02db3a583b4cb545b13d21" exitCode=0 Sep 15 21:43:57.478902 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.478800 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" event={"ID":"f802ab4b-7b3a-47b2-8380-65a77dd4bf87","Type":"ContainerDied","Data":"84befaf220146a9c7830546efa7bb9d4c63e869f3a02db3a583b4cb545b13d21"} Sep 15 21:43:57.480607 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.480585 2559 generic.go:356] "Generic (PLEG): container finished" podID="d51cd259-fa1a-4258-a90c-17249735fc06" containerID="414b5144760eda56a881d147a7e95d1bd3cbd0c3df878ae8b81c9947601cd940" exitCode=0 Sep 15 21:43:57.480740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.480668 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" event={"ID":"d51cd259-fa1a-4258-a90c-17249735fc06","Type":"ContainerDied","Data":"414b5144760eda56a881d147a7e95d1bd3cbd0c3df878ae8b81c9947601cd940"} Sep 15 21:43:57.482152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.482130 2559 generic.go:356] "Generic (PLEG): container finished" podID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" containerID="b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c" exitCode=0 Sep 15 21:43:57.482245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.482174 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" Sep 15 21:43:57.482245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.482214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" event={"ID":"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246","Type":"ContainerDied","Data":"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c"} Sep 15 21:43:57.482340 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.482260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7" event={"ID":"5e29fee9-f4c5-49b9-a2ed-9a2c398a5246","Type":"ContainerDied","Data":"c6b52d953f39ac23bbd5bb55e016e5d61ab2108a9e0a5aceb2fa1ff5185cb6b2"} Sep 15 21:43:57.482340 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.482285 2559 scope.go:117] "RemoveContainer" containerID="b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c" Sep 15 21:43:57.484158 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.484137 2559 generic.go:356] "Generic (PLEG): container finished" podID="950d6cb1-65c7-4a68-854e-79eb133a6e25" containerID="8d92ff5fd39ed4e9638377041ff4b73c8fb2d2c3ca9e05469644dfd64d4a29b5" exitCode=0 Sep 15 21:43:57.484260 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.484199 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" event={"ID":"950d6cb1-65c7-4a68-854e-79eb133a6e25","Type":"ContainerDied","Data":"8d92ff5fd39ed4e9638377041ff4b73c8fb2d2c3ca9e05469644dfd64d4a29b5"} Sep 15 21:43:57.493925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.493907 2559 scope.go:117] "RemoveContainer" containerID="b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c" Sep 15 21:43:57.494284 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:43:57.494168 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c\": container with ID starting with b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c not found: ID does not exist" containerID="b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c" Sep 15 21:43:57.494284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.494211 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c"} err="failed to get container status \"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c\": rpc error: code = NotFound desc = could not find container \"b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c\": container with ID starting with b3572acfee83b5ba3467ecbc626d80784a41a503797a563bf7eee7670410276c not found: ID does not exist" Sep 15 21:43:57.504278 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.504257 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:43:57.510430 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.510406 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:43:57.517498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.517441 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/simulator-w0-b99d7b-kserve-86cc99b4c8-4brq7"] Sep 15 21:43:57.529757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529658 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g76zb\" (UniqueName: \"kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.529757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.529757 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.529987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529824 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.529987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529889 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.529987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.529918 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home\") pod \"d51cd259-fa1a-4258-a90c-17249735fc06\" (UID: \"d51cd259-fa1a-4258-a90c-17249735fc06\") " Sep 15 21:43:57.530145 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.530068 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache" (OuterVolumeSpecName: "model-cache") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.530557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.530228 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.530557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.530263 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.530557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.530438 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home" (OuterVolumeSpecName: "home") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.532576 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.532552 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:57.532932 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.532799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb" (OuterVolumeSpecName: "kube-api-access-g76zb") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "kube-api-access-g76zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.533055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.533034 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm" (OuterVolumeSpecName: "dshm") pod "d51cd259-fa1a-4258-a90c-17249735fc06" (UID: "d51cd259-fa1a-4258-a90c-17249735fc06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.570516 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.570481 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:43:57.608167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.608143 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:43:57.611651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.611630 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:43:57.630599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630576 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630624 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630650 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.630709 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630701 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630719 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630752 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630791 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llgcv\" (UniqueName: \"kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630818 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630841 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630874 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630882 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache" (OuterVolumeSpecName: "model-cache") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630906 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cc6j\" (UniqueName: \"kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630974 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home" (OuterVolumeSpecName: "home") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.630979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.630981 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir\") pod \"950d6cb1-65c7-4a68-854e-79eb133a6e25\" (UID: \"950d6cb1-65c7-4a68-854e-79eb133a6e25\") " Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631043 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p249k\" (UniqueName: \"kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home\") pod \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\" (UID: \"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5\") " Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631140 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm\") pod \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\" (UID: \"f802ab4b-7b3a-47b2-8380-65a77dd4bf87\") " Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631179 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631198 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache" (OuterVolumeSpecName: "model-cache") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631414 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home" (OuterVolumeSpecName: "home") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631750 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631772 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51cd259-fa1a-4258-a90c-17249735fc06-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631786 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631800 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631812 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631826 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g76zb\" (UniqueName: \"kubernetes.io/projected/d51cd259-fa1a-4258-a90c-17249735fc06-kube-api-access-g76zb\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631840 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631856 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631869 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631881 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631895 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.631925 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.631907 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51cd259-fa1a-4258-a90c-17249735fc06-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.632958 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.632551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home" (OuterVolumeSpecName: "home") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.633445 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.633411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.635154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.635125 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j" (OuterVolumeSpecName: "kube-api-access-9cc6j") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "kube-api-access-9cc6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.635764 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.635737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k" (OuterVolumeSpecName: "kube-api-access-p249k") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "kube-api-access-p249k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.635875 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.635809 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm" (OuterVolumeSpecName: "dshm") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.636311 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.636283 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv" (OuterVolumeSpecName: "kube-api-access-llgcv") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "kube-api-access-llgcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:43:57.636540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.636508 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:57.636656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.636636 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm" (OuterVolumeSpecName: "dshm") pod "f802ab4b-7b3a-47b2-8380-65a77dd4bf87" (UID: "f802ab4b-7b3a-47b2-8380-65a77dd4bf87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.638170 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.637577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "950d6cb1-65c7-4a68-854e-79eb133a6e25" (UID: "950d6cb1-65c7-4a68-854e-79eb133a6e25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:57.638170 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.637671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:43:57.638702 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.638680 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm" (OuterVolumeSpecName: "dshm") pod "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" (UID: "aa936ca7-57ed-4f0b-9d38-de5d65e99fe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732829 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p249k\" (UniqueName: \"kubernetes.io/projected/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-kube-api-access-p249k\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732859 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732869 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732878 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950d6cb1-65c7-4a68-854e-79eb133a6e25-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732888 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950d6cb1-65c7-4a68-854e-79eb133a6e25-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732895 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732903 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f802ab4b-7b3a-47b2-8380-65a77dd4bf87-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732911 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732919 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgcv\" (UniqueName: \"kubernetes.io/projected/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-kube-api-access-llgcv\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.732916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732927 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:57.733332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:57.732935 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc6j\" (UniqueName: \"kubernetes.io/projected/950d6cb1-65c7-4a68-854e-79eb133a6e25-kube-api-access-9cc6j\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:43:58.290435 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.290403 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" path="/var/lib/kubelet/pods/5e29fee9-f4c5-49b9-a2ed-9a2c398a5246/volumes" Sep 15 21:43:58.491658 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.491625 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" Sep 15 21:43:58.491922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.491629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk" event={"ID":"aa936ca7-57ed-4f0b-9d38-de5d65e99fe5","Type":"ContainerDied","Data":"316bc5c89280c9f598cad4ebdf38d85d6e6a42b86f4040b40dfd28a7931d2b29"} Sep 15 21:43:58.491922 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.491766 2559 scope.go:117] "RemoveContainer" containerID="99733f38d469198f0ede59ffbd18a7134b5317a99ca064ab9940c44b2b703f47" Sep 15 21:43:58.493139 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.493117 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" Sep 15 21:43:58.493262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.493119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j" event={"ID":"f802ab4b-7b3a-47b2-8380-65a77dd4bf87","Type":"ContainerDied","Data":"3eb6dfeb6e3a2c5984ab9ce3b10cea5f77ba4cf527d24443f81ff44573e60a18"} Sep 15 21:43:58.494870 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.494850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" event={"ID":"d51cd259-fa1a-4258-a90c-17249735fc06","Type":"ContainerDied","Data":"3f8c4283bf1d21140b1c941885cfcd286baed90db8f22d270f8daf616001e8d5"} Sep 15 21:43:58.494987 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.494878 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6" Sep 15 21:43:58.497877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.497827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" event={"ID":"950d6cb1-65c7-4a68-854e-79eb133a6e25","Type":"ContainerDied","Data":"ddd9dfda9507dc096fe984dd33730ac8cd333dcd67fea8124b51fe7ca01222b1"} Sep 15 21:43:58.497877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.497871 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf" Sep 15 21:43:58.502399 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.502383 2559 scope.go:117] "RemoveContainer" containerID="84befaf220146a9c7830546efa7bb9d4c63e869f3a02db3a583b4cb545b13d21" Sep 15 21:43:58.509776 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.509760 2559 scope.go:117] "RemoveContainer" containerID="414b5144760eda56a881d147a7e95d1bd3cbd0c3df878ae8b81c9947601cd940" Sep 15 21:43:58.517713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.517698 2559 scope.go:117] "RemoveContainer" containerID="8d92ff5fd39ed4e9638377041ff4b73c8fb2d2c3ca9e05469644dfd64d4a29b5" Sep 15 21:43:58.518694 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.518679 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:43:58.523248 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.523228 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/premium-w4-ac4411-kserve-577bdd8fb6-xj4xf"] Sep 15 21:43:58.534497 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.534475 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:43:58.538213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.538192 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/simulator-w4-ac4411-kserve-8586c556c5-spjjk"] Sep 15 21:43:58.548545 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.548499 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:43:58.552685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.552665 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-w4-ac4411-kserve-5fb4777945-2v57j"] Sep 15 21:43:58.565229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.565210 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:43:58.567207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:58.567190 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/distinct-2-w4-ac4411-kserve-544c89554b-7j4j6"] Sep 15 21:43:59.057523 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.057484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:59.158376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.158342 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:59.309821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.309734 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:43:59.310077 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.310016 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" containerID="cri-o://0e0dccca693ba20c3dde3eeb19f0e4968d90ccfb1c430e579b4cb0120bc64cb6" gracePeriod=30 Sep 15 21:43:59.346756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.346719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:43:59.786366 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.786319 2559 patch_prober.go:28] interesting pod/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:43:59.786535 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:43:59.786400 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:44:00.291371 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.291328 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950d6cb1-65c7-4a68-854e-79eb133a6e25" path="/var/lib/kubelet/pods/950d6cb1-65c7-4a68-854e-79eb133a6e25/volumes" Sep 15 21:44:00.291939 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.291917 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" path="/var/lib/kubelet/pods/aa936ca7-57ed-4f0b-9d38-de5d65e99fe5/volumes" Sep 15 21:44:00.292351 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.292332 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51cd259-fa1a-4258-a90c-17249735fc06" path="/var/lib/kubelet/pods/d51cd259-fa1a-4258-a90c-17249735fc06/volumes" Sep 15 21:44:00.293057 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.293035 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" path="/var/lib/kubelet/pods/f802ab4b-7b3a-47b2-8380-65a77dd4bf87/volumes" Sep 15 21:44:00.375166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.375134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:00.474539 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.474508 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:00.637905 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:00.637876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:04.535143 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.535115 2559 generic.go:356] "Generic (PLEG): container finished" podID="677b8361-c00d-4104-86f9-651ce8b839da" containerID="0e0dccca693ba20c3dde3eeb19f0e4968d90ccfb1c430e579b4cb0120bc64cb6" exitCode=0 Sep 15 21:44:04.535491 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.535152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerDied","Data":"0e0dccca693ba20c3dde3eeb19f0e4968d90ccfb1c430e579b4cb0120bc64cb6"} Sep 15 21:44:04.535491 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.535227 2559 scope.go:117] "RemoveContainer" containerID="a9bf09c12280b109f84160dcd7c784072aa974e60bb73727d3591fe66c794edd" Sep 15 21:44:04.583824 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.583801 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:44:04.599980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.599914 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.599980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.599967 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600071 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600097 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600131 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxb64\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600157 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:04.600370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600271 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token\") pod \"677b8361-c00d-4104-86f9-651ce8b839da\" (UID: \"677b8361-c00d-4104-86f9-651ce8b839da\") " Sep 15 21:44:04.600595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600514 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data" (OuterVolumeSpecName: "istio-data") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:04.600595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:44:04.600754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600723 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:04.600754 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600739 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.600884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600761 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677b8361-c00d-4104-86f9-651ce8b839da-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.600884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600766 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:04.600884 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.600775 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.602149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.602124 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:04.602366 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.602347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64" (OuterVolumeSpecName: "kube-api-access-rxb64") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "kube-api-access-rxb64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:04.602683 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.602654 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token" (OuterVolumeSpecName: "istio-token") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:04.602755 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.602689 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "677b8361-c00d-4104-86f9-651ce8b839da" (UID: "677b8361-c00d-4104-86f9-651ce8b839da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:44:04.701517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701494 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.701517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701515 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.701662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701525 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxb64\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-kube-api-access-rxb64\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.701662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701535 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677b8361-c00d-4104-86f9-651ce8b839da-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.701662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701543 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677b8361-c00d-4104-86f9-651ce8b839da-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:04.701662 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:04.701550 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677b8361-c00d-4104-86f9-651ce8b839da-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:05.541786 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:05.541759 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" Sep 15 21:44:05.542206 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:05.541759 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz" event={"ID":"677b8361-c00d-4104-86f9-651ce8b839da","Type":"ContainerDied","Data":"4b7b1a227fd0202ccb0a6eb4b8d303e457b1fe2f39db744a259325bda8bf663e"} Sep 15 21:44:05.542206 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:05.541913 2559 scope.go:117] "RemoveContainer" containerID="0e0dccca693ba20c3dde3eeb19f0e4968d90ccfb1c430e579b4cb0120bc64cb6" Sep 15 21:44:05.565440 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:05.565408 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:44:05.567644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:05.567623 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b99d7b-openshift-default-b99cd66b6-vt6hz"] Sep 15 21:44:06.290403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:06.290371 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677b8361-c00d-4104-86f9-651ce8b839da" path="/var/lib/kubelet/pods/677b8361-c00d-4104-86f9-651ce8b839da/volumes" Sep 15 21:44:14.914978 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:14.914891 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:44:18.888214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:18.888180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:19.021842 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:19.021796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:20.524252 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:20.524218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:21.012833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:21.012779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:21.113557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:21.113508 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:21.215787 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:21.215752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:26.359527 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:26.359474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:26.717001 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:26.716903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:28.423819 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:28.423781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:29.328619 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:29.328582 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:44:30.219931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:30.219897 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:44:30.245799 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:30.245775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:32.206078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:32.206045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:32.599405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:32.599372 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:36.158353 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:36.158322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:38.629405 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:38.629367 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:44:45.239813 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.239773 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" podUID="e49a0892-1e9b-444f-baca-f4c38bded9c6" containerName="main" containerID="cri-o://f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65" gracePeriod=45 Sep 15 21:44:45.491048 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.490993 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:44:45.659372 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl58p\" (UniqueName: \"kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659406 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659476 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659514 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659558 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm\") pod \"e49a0892-1e9b-444f-baca-f4c38bded9c6\" (UID: \"e49a0892-1e9b-444f-baca-f4c38bded9c6\") " Sep 15 21:44:45.659848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659677 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:45.659848 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home" (OuterVolumeSpecName: "home") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:45.659926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659856 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.659926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.659873 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.660099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.660070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:45.661828 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.661801 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:44:45.661828 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.661816 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm" (OuterVolumeSpecName: "dshm") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:44:45.661998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.661919 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p" (OuterVolumeSpecName: "kube-api-access-dl58p") pod "e49a0892-1e9b-444f-baca-f4c38bded9c6" (UID: "e49a0892-1e9b-444f-baca-f4c38bded9c6"). InnerVolumeSpecName "kube-api-access-dl58p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:44:45.744107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.744019 2559 generic.go:356] "Generic (PLEG): container finished" podID="e49a0892-1e9b-444f-baca-f4c38bded9c6" containerID="f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65" exitCode=0 Sep 15 21:44:45.744107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.744065 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" Sep 15 21:44:45.744107 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.744097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" event={"ID":"e49a0892-1e9b-444f-baca-f4c38bded9c6","Type":"ContainerDied","Data":"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65"} Sep 15 21:44:45.744374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.744138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn" event={"ID":"e49a0892-1e9b-444f-baca-f4c38bded9c6","Type":"ContainerDied","Data":"a1c2b724612d541214176367f78ba462470ef4c90866fe9128c92fbf5c883675"} Sep 15 21:44:45.744374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.744155 2559 scope.go:117] "RemoveContainer" containerID="f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65" Sep 15 21:44:45.753322 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.753305 2559 scope.go:117] "RemoveContainer" containerID="f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65" Sep 15 21:44:45.753625 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:44:45.753607 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65\": container with ID starting with f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65 not found: ID does not exist" containerID="f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65" Sep 15 21:44:45.753679 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.753633 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65"} err="failed to get container status \"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65\": rpc error: code = NotFound desc = could not find container \"f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65\": container with ID starting with f03095670df13cb6c6042a4e0b9497e4a7411fc7bd88f91ecdef7ab64625cf65 not found: ID does not exist" Sep 15 21:44:45.760548 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.760526 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.760548 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.760549 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49a0892-1e9b-444f-baca-f4c38bded9c6-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.760688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.760557 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49a0892-1e9b-444f-baca-f4c38bded9c6-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.760688 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.760565 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl58p\" (UniqueName: \"kubernetes.io/projected/e49a0892-1e9b-444f-baca-f4c38bded9c6-kube-api-access-dl58p\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:44:45.765969 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.765947 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:44:45.769615 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:45.769592 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/auth-test-model-38a3a3-kserve-86b946d575-xvlnn"] Sep 15 21:44:46.290117 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:46.290084 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49a0892-1e9b-444f-baca-f4c38bded9c6" path="/var/lib/kubelet/pods/e49a0892-1e9b-444f-baca-f4c38bded9c6/volumes" Sep 15 21:44:52.510546 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:52.510515 2559 scope.go:117] "RemoveContainer" containerID="fda55678b441894f77e60f17fa05fd1aba960f274c89ab3194cfd78402961c23" Sep 15 21:44:56.285184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:56.285150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:44:56.384958 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:44:56.384922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:05.318402 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318370 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318717 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318729 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318738 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51cd259-fa1a-4258-a90c-17249735fc06" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318743 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51cd259-fa1a-4258-a90c-17249735fc06" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318752 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318758 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318765 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318770 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318777 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49a0892-1e9b-444f-baca-f4c38bded9c6" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318782 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49a0892-1e9b-444f-baca-f4c38bded9c6" containerName="main" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318791 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.318789 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318796 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318805 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318810 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318818 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263fd6e8-fb5b-4be5-b840-b8c204e63ce3" containerName="revoke-keys" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318822 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="263fd6e8-fb5b-4be5-b840-b8c204e63ce3" containerName="revoke-keys" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318831 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950d6cb1-65c7-4a68-854e-79eb133a6e25" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318837 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="950d6cb1-65c7-4a68-854e-79eb133a6e25" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318890 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51cd259-fa1a-4258-a90c-17249735fc06" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318898 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa936ca7-57ed-4f0b-9d38-de5d65e99fe5" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318904 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318911 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49a0892-1e9b-444f-baca-f4c38bded9c6" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318916 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="950d6cb1-65c7-4a68-854e-79eb133a6e25" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318922 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="677b8361-c00d-4104-86f9-651ce8b839da" containerName="istio-proxy" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318927 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="263fd6e8-fb5b-4be5-b840-b8c204e63ce3" containerName="revoke-keys" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318932 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f802ab4b-7b3a-47b2-8380-65a77dd4bf87" containerName="main" Sep 15 21:45:05.319166 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.318938 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e29fee9-f4c5-49b9-a2ed-9a2c398a5246" containerName="main" Sep 15 21:45:05.321807 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.321783 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.324781 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.324756 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"kube-root-ca.crt\"" Sep 15 21:45:05.325950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.325929 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"default-dockercfg-xszmj\"" Sep 15 21:45:05.325950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.325935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"openshift-service-ca.crt\"" Sep 15 21:45:05.326120 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.325967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"sub-test-model-38a3a3-kserve-self-signed-certs\"" Sep 15 21:45:05.332448 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.332427 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:45:05.348883 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.348858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:05.430880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.430844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tx8h\" (UniqueName: \"kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.431051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.430885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.431051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.430942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.431051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.431019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.431051 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.431038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.431192 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.431056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532480 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532522 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tx8h\" (UniqueName: \"kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.532869 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.532743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.533037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.533008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.533121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.533046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.533121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.533084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.534957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.534928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.535232 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.535212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.540591 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.540566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tx8h\" (UniqueName: \"kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h\") pod \"sub-test-model-38a3a3-kserve-57b4c64957-f5sq6\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.631415 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.631382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:05.904833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:05.904677 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:45:05.907160 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:45:05.907125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd130aa4b_9c11_453e_9f0a_613c6588632c.slice/crio-fadfc6dc060a3180556e7609170938c5b3c69351788c5593062c8c2e35ccb88c WatchSource:0}: Error finding container fadfc6dc060a3180556e7609170938c5b3c69351788c5593062c8c2e35ccb88c: Status 404 returned error can't find the container with id fadfc6dc060a3180556e7609170938c5b3c69351788c5593062c8c2e35ccb88c Sep 15 21:45:06.851525 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:06.851486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" event={"ID":"d130aa4b-9c11-453e-9f0a-613c6588632c","Type":"ContainerStarted","Data":"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3"} Sep 15 21:45:06.851906 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:06.851538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" event={"ID":"d130aa4b-9c11-453e-9f0a-613c6588632c","Type":"ContainerStarted","Data":"fadfc6dc060a3180556e7609170938c5b3c69351788c5593062c8c2e35ccb88c"} Sep 15 21:45:06.873529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:06.873481 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" podStartSLOduration=1.873450367 podStartE2EDuration="1.873450367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:06.872380299 +0000 UTC m=+1755.097003303" watchObservedRunningTime="2026-09-15 21:45:06.873450367 +0000 UTC m=+1755.098073370" Sep 15 21:45:08.926857 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:08.926824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:45:15.632058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:15.631963 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:15.632581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:15.632094 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:15.644553 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:15.644529 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:15.908800 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:15.908720 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:45:22.717130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:22.717083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:24.130288 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:24.130252 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:45:36.918488 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:36.918442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:45:39.626616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.626584 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7754-c6db83"] Sep 15 21:45:39.630023 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.630006 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:39.632820 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.632799 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\"" Sep 15 21:45:39.636926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.636899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-c6db83"] Sep 15 21:45:39.719656 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.719625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm42x\" (UniqueName: \"kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x\") pod \"test-curl-7754-c6db83\" (UID: \"4ba53704-4b35-4a0b-b116-123b5830ed5b\") " pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:39.820632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.820600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm42x\" (UniqueName: \"kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x\") pod \"test-curl-7754-c6db83\" (UID: \"4ba53704-4b35-4a0b-b116-123b5830ed5b\") " pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:39.830693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.830671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm42x\" (UniqueName: \"kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x\") pod \"test-curl-7754-c6db83\" (UID: \"4ba53704-4b35-4a0b-b116-123b5830ed5b\") " pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:39.940403 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:39.940327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:40.208856 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:40.208830 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-c6db83"] Sep 15 21:45:40.211184 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:45:40.211149 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ba53704_4b35_4a0b_b116_123b5830ed5b.slice/crio-eb0090e9724edb5d46dd3792a2ea75260a3dc8aa1f18849f1bb0baedac782d80 WatchSource:0}: Error finding container eb0090e9724edb5d46dd3792a2ea75260a3dc8aa1f18849f1bb0baedac782d80: Status 404 returned error can't find the container with id eb0090e9724edb5d46dd3792a2ea75260a3dc8aa1f18849f1bb0baedac782d80 Sep 15 21:45:41.022978 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:41.022943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-c6db83" event={"ID":"4ba53704-4b35-4a0b-b116-123b5830ed5b","Type":"ContainerStarted","Data":"a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677"} Sep 15 21:45:41.023352 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:41.022984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-c6db83" event={"ID":"4ba53704-4b35-4a0b-b116-123b5830ed5b","Type":"ContainerStarted","Data":"eb0090e9724edb5d46dd3792a2ea75260a3dc8aa1f18849f1bb0baedac782d80"} Sep 15 21:45:41.048410 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:41.048360 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7754-c6db83" podStartSLOduration=2.048344378 podStartE2EDuration="2.048344378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:41.045408962 +0000 UTC m=+1789.270031966" watchObservedRunningTime="2026-09-15 21:45:41.048344378 +0000 UTC m=+1789.272967381" Sep 15 21:45:41.626903 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:41.626865 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7754-c6db83"] Sep 15 21:45:41.630898 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:41.630873 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7754-c6db83"] Sep 15 21:45:42.027287 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:45:42.027255 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7754-c6db83" container="test-curl-7754-c6db83" Sep 15 21:45:43.032308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:43.032269 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7754-c6db83" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" containerName="test-curl-7754-c6db83" containerID="cri-o://a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677" gracePeriod=2 Sep 15 21:45:45.220773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:45.220749 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:45.229377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:45.229351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:45.268769 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:45.268743 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm42x\" (UniqueName: \"kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x\") pod \"4ba53704-4b35-4a0b-b116-123b5830ed5b\" (UID: \"4ba53704-4b35-4a0b-b116-123b5830ed5b\") " Sep 15 21:45:45.270706 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:45.270683 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x" (OuterVolumeSpecName: "kube-api-access-hm42x") pod "4ba53704-4b35-4a0b-b116-123b5830ed5b" (UID: "4ba53704-4b35-4a0b-b116-123b5830ed5b"). InnerVolumeSpecName "kube-api-access-hm42x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:45:45.369632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:45.369593 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm42x\" (UniqueName: \"kubernetes.io/projected/4ba53704-4b35-4a0b-b116-123b5830ed5b-kube-api-access-hm42x\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:45:46.048058 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.048020 2559 generic.go:356] "Generic (PLEG): container finished" podID="4ba53704-4b35-4a0b-b116-123b5830ed5b" containerID="a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677" exitCode=137 Sep 15 21:45:46.048250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.048063 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:46.048250 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.048131 2559 scope.go:117] "RemoveContainer" containerID="a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677" Sep 15 21:45:46.056699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.056682 2559 scope.go:117] "RemoveContainer" containerID="a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677" Sep 15 21:45:46.056973 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:45:46.056953 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677\": container with ID starting with a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677 not found: ID does not exist" containerID="a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677" Sep 15 21:45:46.057026 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.056983 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677"} err="failed to get container status \"a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677\": rpc error: code = NotFound desc = could not find container \"a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677\": container with ID starting with a8d07476152bfaf33b2da8712ff69eaa1c77d281f993a456a23382dec85ee677 not found: ID does not exist" Sep 15 21:45:46.059505 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:45:46.059448 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7754-c6db83\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:46.128722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.128690 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:45:46.148271 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:45:46.148243 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7754-c6db83\" is forbidden: User \"system:node:ip-10-0-129-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-100.ec2.internal' and this object" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" pod="opendatahub/test-curl-7754-c6db83" Sep 15 21:45:46.150700 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.150680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:46.290597 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:46.290571 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" path="/var/lib/kubelet/pods/4ba53704-4b35-4a0b-b116-123b5830ed5b/volumes" Sep 15 21:45:48.015167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:48.015135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:45:49.718224 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:49.718194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:45:59.215041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:45:59.215007 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:46:01.147560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.147522 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" podUID="d130aa4b-9c11-453e-9f0a-613c6588632c" containerName="main" containerID="cri-o://134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3" gracePeriod=45 Sep 15 21:46:01.399635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.399572 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:46:01.507660 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.507822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507683 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.507822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.507822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.507822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tx8h\" (UniqueName: \"kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.508064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507832 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache\") pod \"d130aa4b-9c11-453e-9f0a-613c6588632c\" (UID: \"d130aa4b-9c11-453e-9f0a-613c6588632c\") " Sep 15 21:46:01.508064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home" (OuterVolumeSpecName: "home") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:46:01.508064 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.507990 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:46:01.508214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.508137 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:01.508214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.508164 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:01.508214 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.508140 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:46:01.509822 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.509800 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h" (OuterVolumeSpecName: "kube-api-access-8tx8h") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "kube-api-access-8tx8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:46:01.510263 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.510245 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:46:01.510319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.510255 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm" (OuterVolumeSpecName: "dshm") pod "d130aa4b-9c11-453e-9f0a-613c6588632c" (UID: "d130aa4b-9c11-453e-9f0a-613c6588632c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:46:01.608847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.608813 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tx8h\" (UniqueName: \"kubernetes.io/projected/d130aa4b-9c11-453e-9f0a-613c6588632c-kube-api-access-8tx8h\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:01.608847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.608844 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:01.608847 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.608854 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d130aa4b-9c11-453e-9f0a-613c6588632c-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:01.609059 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:01.608862 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d130aa4b-9c11-453e-9f0a-613c6588632c-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:46:02.127282 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.127241 2559 generic.go:356] "Generic (PLEG): container finished" podID="d130aa4b-9c11-453e-9f0a-613c6588632c" containerID="134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3" exitCode=0 Sep 15 21:46:02.127282 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.127285 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" Sep 15 21:46:02.127581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.127342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" event={"ID":"d130aa4b-9c11-453e-9f0a-613c6588632c","Type":"ContainerDied","Data":"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3"} Sep 15 21:46:02.127581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.127390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6" event={"ID":"d130aa4b-9c11-453e-9f0a-613c6588632c","Type":"ContainerDied","Data":"fadfc6dc060a3180556e7609170938c5b3c69351788c5593062c8c2e35ccb88c"} Sep 15 21:46:02.127581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.127414 2559 scope.go:117] "RemoveContainer" containerID="134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3" Sep 15 21:46:02.144879 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.144861 2559 scope.go:117] "RemoveContainer" containerID="134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3" Sep 15 21:46:02.145132 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:46:02.145114 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3\": container with ID starting with 134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3 not found: ID does not exist" containerID="134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3" Sep 15 21:46:02.145190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.145140 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3"} err="failed to get container status \"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3\": rpc error: code = NotFound desc = could not find container \"134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3\": container with ID starting with 134e4538a30fee7abd3b32cde0789a7a1b770c2cda607f2e3c63154d3c6696e3 not found: ID does not exist" Sep 15 21:46:02.154719 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.154690 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:46:02.161770 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.161739 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/sub-test-model-38a3a3-kserve-57b4c64957-f5sq6"] Sep 15 21:46:02.290330 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:02.290303 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d130aa4b-9c11-453e-9f0a-613c6588632c" path="/var/lib/kubelet/pods/d130aa4b-9c11-453e-9f0a-613c6588632c/volumes" Sep 15 21:46:19.986228 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:19.986188 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:23.851735 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:23.851694 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:46:38.921128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:38.921091 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:46:49.525133 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:49.525097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:46:54.669873 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:54.669836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:56.348843 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:56.348810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:56.524114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:56.524073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:56.672957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:56.672874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:59.358280 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:59.358245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:59.477294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:59.477261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:46:59.569742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:46:59.569706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:00.369936 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:00.369899 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.240231 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240198 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:47:01.240657 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240639 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d130aa4b-9c11-453e-9f0a-613c6588632c" containerName="main" Sep 15 21:47:01.240773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240659 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d130aa4b-9c11-453e-9f0a-613c6588632c" containerName="main" Sep 15 21:47:01.240773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240690 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" containerName="test-curl-7754-c6db83" Sep 15 21:47:01.240773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240699 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" containerName="test-curl-7754-c6db83" Sep 15 21:47:01.240927 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240795 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba53704-4b35-4a0b-b116-123b5830ed5b" containerName="test-curl-7754-c6db83" Sep 15 21:47:01.240927 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.240810 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d130aa4b-9c11-453e-9f0a-613c6588632c" containerName="main" Sep 15 21:47:01.243992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.243971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.246544 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.246519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b2c5fe-serving-cert\"" Sep 15 21:47:01.255068 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.255043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:47:01.302560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.302532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.302687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.302579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.302687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.302603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jrf\" (UniqueName: \"kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.302687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.302665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.312043 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.312020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.403918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.403877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.404377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.403951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.404377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.403989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jrf\" (UniqueName: \"kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.404377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.404024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.404377 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:47:01.404124 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-serving-cert: secret "maas-api-e2e-ipp-a-b2c5fe-serving-cert" not found Sep 15 21:47:01.404377 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:47:01.404211 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls podName:d8c6b381-a0ed-49b1-aaf2-cec011c62b82 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:01.904188085 +0000 UTC m=+1870.128811072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls") pod "maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" (UID: "d8c6b381-a0ed-49b1-aaf2-cec011c62b82") : secret "maas-api-e2e-ipp-a-b2c5fe-serving-cert" not found Sep 15 21:47:01.404693 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.404673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.406632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.406604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.413171 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.413145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.415450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.415424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jrf\" (UniqueName: \"kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.612161 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.612128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.775362 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.775330 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.875101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.875025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:01.908367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.908342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:01.910798 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:01.910777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:02.155984 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:02.155896 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:02.428967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:02.428941 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:47:02.431061 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:47:02.431038 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c6b381_a0ed_49b1_aaf2_cec011c62b82.slice/crio-d526e48edee471db6bbf6800cb440793433ba19a0e2f2dc4fa069816643bbcd0 WatchSource:0}: Error finding container d526e48edee471db6bbf6800cb440793433ba19a0e2f2dc4fa069816643bbcd0: Status 404 returned error can't find the container with id d526e48edee471db6bbf6800cb440793433ba19a0e2f2dc4fa069816643bbcd0 Sep 15 21:47:02.432638 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:02.432615 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:47:03.428035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:03.427943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" event={"ID":"d8c6b381-a0ed-49b1-aaf2-cec011c62b82","Type":"ContainerStarted","Data":"928e3736db8a93b1e9746a61043649999d261bb9efc3209cf31ad37319063974"} Sep 15 21:47:03.428035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:03.428003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" event={"ID":"d8c6b381-a0ed-49b1-aaf2-cec011c62b82","Type":"ContainerStarted","Data":"d526e48edee471db6bbf6800cb440793433ba19a0e2f2dc4fa069816643bbcd0"} Sep 15 21:47:03.428035 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:03.428034 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:03.448685 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:03.448634 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" podStartSLOduration=1.794611521 podStartE2EDuration="2.448617733s" totalImagesPullingTime="654.006212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:03.446620572 +0000 UTC m=+1871.671243575" watchObservedRunningTime="2026-09-15 21:47:03.448617733 +0000 UTC m=+1871.673240737" Sep 15 21:47:04.723722 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:04.723687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:47:08.470367 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:08.470337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:09.438465 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:09.438426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:47:23.616557 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:23.616479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:25.025645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:25.025605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:25.872517 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:25.872479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:26.672297 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:26.672258 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:27.617220 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:27.617187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:29.559260 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:29.559231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:30.208299 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:30.208267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:30.523991 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:30.523909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:30.628915 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:30.628874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:32.553264 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:32.553234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:33.172255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:33.172223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:33.271446 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:33.271411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:33.372586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:33.372556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:34.325955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:34.325922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:35.078534 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:35.078504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:35.970774 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:35.970737 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:36.072065 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:36.072035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:36.465236 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:36.465208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:36.565332 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:36.565302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:37.414654 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:37.414620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:38.899661 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:38.899628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:38.953973 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:38.953948 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:40.019612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:40.019579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:47:46.173668 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:46.173635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:57.949697 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:57.949666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:47:59.355839 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:47:59.355779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:00.354174 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:00.354133 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:01.085829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:01.085798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:01.816957 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:01.816927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:02.684623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:02.684591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:03.247172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:03.247139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:22.214165 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:22.214129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:28.826388 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.826356 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:48:28.829737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.829717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.834109 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.834082 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b2c5fe\"/\"ipp-route-b2c5fe-kserve-self-signed-certs\"" Sep 15 21:48:28.834245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.834091 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b2c5fe\"/\"openshift-service-ca.crt\"" Sep 15 21:48:28.834245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.834092 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b2c5fe\"/\"default-dockercfg-lgqp9\"" Sep 15 21:48:28.834245 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.834093 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b2c5fe\"/\"kube-root-ca.crt\"" Sep 15 21:48:28.843184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843163 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:48:28.843285 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.843344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.843344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.843344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.843481 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sh69\" (UniqueName: \"kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.843481 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.843412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.878542 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.878511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:28.943933 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.943905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944122 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.943966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944122 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh69\" (UniqueName: \"kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944395 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944438 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.944547 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.944527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.946186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.946165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.946614 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.946593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:28.953125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:28.953100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh69\" (UniqueName: \"kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69\") pod \"ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:29.140864 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:29.140782 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:29.416979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:29.416895 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:48:29.420105 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:48:29.420078 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8094d621_69f2_43c3_9baf_daa208ffe48b.slice/crio-dad666dbca478b155248c2f6a0741312f32cd8a7fe3b8fd352558355e35f6414 WatchSource:0}: Error finding container dad666dbca478b155248c2f6a0741312f32cd8a7fe3b8fd352558355e35f6414: Status 404 returned error can't find the container with id dad666dbca478b155248c2f6a0741312f32cd8a7fe3b8fd352558355e35f6414 Sep 15 21:48:29.839090 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:29.839054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" event={"ID":"8094d621-69f2-43c3-9baf-daa208ffe48b","Type":"ContainerStarted","Data":"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f"} Sep 15 21:48:29.839495 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:29.839107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" event={"ID":"8094d621-69f2-43c3-9baf-daa208ffe48b","Type":"ContainerStarted","Data":"dad666dbca478b155248c2f6a0741312f32cd8a7fe3b8fd352558355e35f6414"} Sep 15 21:48:29.856762 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:29.856708 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" podStartSLOduration=1.856691694 podStartE2EDuration="1.856691694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:48:29.856614057 +0000 UTC m=+1958.081237065" watchObservedRunningTime="2026-09-15 21:48:29.856691694 +0000 UTC m=+1958.081314696" Sep 15 21:48:33.050251 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:33.050215 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:48:39.141686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:39.141654 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:39.142132 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:39.141884 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:39.154375 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:39.154352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:39.901790 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:39.901764 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:48:47.929094 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:47.929004 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:48:54.192066 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:54.192030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:54.294141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:54.294105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:54.820005 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:54.819947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:55.108887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.108856 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:48:55.112514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.112493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.115678 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.115659 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-5dc07b-openshift-default-dockercfg-s6tpb\"" Sep 15 21:48:55.128569 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.128547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:48:55.146687 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.146649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:55.182373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182347 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182506 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtzq\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.182855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.182729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.283781 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.283751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.283794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtzq\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.283817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.283967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284240 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284285 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.284861 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.284696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.286496 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.286450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.286731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.286712 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.292961 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.292936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtzq\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.293644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.293629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token\") pod \"e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.424286 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.424223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:55.501972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.501944 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:55.702765 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.702737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:48:55.703812 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:48:55.703788 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd274e6_72e8_401c_958b_c7ef1f152273.slice/crio-a08717c4e7bf3696dab32bb5c1e21b16c73c4c87e790d4ca7896d3c7b60d4071 WatchSource:0}: Error finding container a08717c4e7bf3696dab32bb5c1e21b16c73c4c87e790d4ca7896d3c7b60d4071: Status 404 returned error can't find the container with id a08717c4e7bf3696dab32bb5c1e21b16c73c4c87e790d4ca7896d3c7b60d4071 Sep 15 21:48:55.706316 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.706275 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:48:55.706383 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.706361 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:48:55.706424 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.706402 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:48:55.973836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.973734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerStarted","Data":"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5"} Sep 15 21:48:55.973836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.973832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerStarted","Data":"a08717c4e7bf3696dab32bb5c1e21b16c73c4c87e790d4ca7896d3c7b60d4071"} Sep 15 21:48:55.996584 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:55.996538 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podStartSLOduration=0.996523129 podStartE2EDuration="996.523129ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:48:55.995526462 +0000 UTC m=+1984.220149465" watchObservedRunningTime="2026-09-15 21:48:55.996523129 +0000 UTC m=+1984.221146133" Sep 15 21:48:56.424532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:56.424501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:56.429444 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:56.429419 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:56.978493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:56.978436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:56.979278 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:56.979262 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:48:57.265163 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:57.265094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:57.916897 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:57.916866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:58.025971 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:58.025932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:58.299376 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:58.299291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:48:59.378039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:48:59.378000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:00.190932 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:00.190903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:01.003510 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:01.003481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:02.032972 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:02.032937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:02.140504 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:02.140472 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:02.232492 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:02.232438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:02.587409 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:02.587370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:02.686846 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:02.686797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:07.432830 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:07.432793 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:49:18.083041 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.083010 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerID="676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5" exitCode=0 Sep 15 21:49:18.083467 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.083084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerDied","Data":"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5"} Sep 15 21:49:18.083563 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.083545 2559 scope.go:117] "RemoveContainer" containerID="676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5" Sep 15 21:49:18.084674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.084645 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:49:18.084738 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.084718 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:49:18.084774 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:18.084749 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 21:49:19.090545 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:19.090508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerStarted","Data":"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85"} Sep 15 21:49:19.425391 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:19.425309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:19.425572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:19.425501 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:19.425572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:19.425544 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:20.424733 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:20.424699 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:20.425121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:20.424767 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:21.424643 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:21.424610 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:21.424809 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:21.424698 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:22.425515 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:22.425477 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:22.425900 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:22.425545 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:23.327096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:23.327062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:23.425508 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:23.425472 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:23.425716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:23.425542 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:23.427975 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:23.427947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:24.283055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:24.283014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:24.424734 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:24.424696 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:49:24.424924 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:24.424765 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 15 21:49:25.188000 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:25.187967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:25.429324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:25.429292 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:25.429606 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:25.429580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:25.430216 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:25.430199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:25.968423 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:25.968394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:26.958953 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:26.958924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:27.794183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:27.794150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:28.047387 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:28.047298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:28.611998 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:28.611969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:29.732122 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:29.732092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:30.558472 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:30.558421 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:31.480647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:31.480606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:32.335482 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:32.335439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:33.227568 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:33.227536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:33.983037 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:33.983007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:39.880094 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:39.880060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:39.982213 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:39.982182 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:40.082623 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:40.082593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:40.148909 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:40.148835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:40.479943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:40.479851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:40.601878 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:40.601849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:41.485185 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:41.485149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:42.363354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:42.363323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:43.112921 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:43.112884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:43.872468 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:43.872437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:44.327347 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:44.327319 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:49:46.648646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:46.648613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:46.747931 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:46.747901 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:46.878756 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:46.878723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:49:46.879019 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:46.878990 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" containerID="cri-o://8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85" gracePeriod=30 Sep 15 21:49:46.902729 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:46.902660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:47.235645 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:47.235567 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:47.312864 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:47.312837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:50.620370 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:50.620339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:51.520122 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:51.520090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:52.126253 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.126229 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:52.273650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.273574 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerID="8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85" exitCode=0 Sep 15 21:49:52.273785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.273652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" Sep 15 21:49:52.273785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.273664 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerDied","Data":"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85"} Sep 15 21:49:52.273785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.273745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct" event={"ID":"6bd274e6-72e8-401c-958b-c7ef1f152273","Type":"ContainerDied","Data":"a08717c4e7bf3696dab32bb5c1e21b16c73c4c87e790d4ca7896d3c7b60d4071"} Sep 15 21:49:52.273785 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.273769 2559 scope.go:117] "RemoveContainer" containerID="8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85" Sep 15 21:49:52.282298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.282283 2559 scope.go:117] "RemoveContainer" containerID="676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5" Sep 15 21:49:52.290125 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.290107 2559 scope.go:117] "RemoveContainer" containerID="8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85" Sep 15 21:49:52.290397 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:49:52.290378 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85\": container with ID starting with 8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85 not found: ID does not exist" containerID="8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85" Sep 15 21:49:52.290493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.290404 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85"} err="failed to get container status \"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85\": rpc error: code = NotFound desc = could not find container \"8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85\": container with ID starting with 8c00502ba577ea1672c8aae4a12b9d2d3ee9957a036fcbd93f43f0e5c8ceab85 not found: ID does not exist" Sep 15 21:49:52.290493 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.290422 2559 scope.go:117] "RemoveContainer" containerID="676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5" Sep 15 21:49:52.290674 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:49:52.290656 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5\": container with ID starting with 676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5 not found: ID does not exist" containerID="676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5" Sep 15 21:49:52.290713 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.290677 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5"} err="failed to get container status \"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5\": rpc error: code = NotFound desc = could not find container \"676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5\": container with ID starting with 676e2687934bb1875be4f146e4aac1de219eee58c6d30a03cae4cb49c75850c5 not found: ID does not exist" Sep 15 21:49:52.307529 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307507 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xtzq\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307539 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307561 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307618 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307597 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307779 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307626 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307779 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307655 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307779 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307693 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307779 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307719 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.307779 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert\") pod \"6bd274e6-72e8-401c-958b-c7ef1f152273\" (UID: \"6bd274e6-72e8-401c-958b-c7ef1f152273\") " Sep 15 21:49:52.308039 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307954 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:49:52.308100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308047 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:49:52.308100 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.307962 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:49:52.308207 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308183 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:49:52.308294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308263 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:49:52.308294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308283 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-credential-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.308412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308304 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-data\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.308412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308318 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.308412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.308332 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-workload-socket\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.309674 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.309646 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:49:52.309854 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.309831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:49:52.310015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.309995 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq" (OuterVolumeSpecName: "kube-api-access-6xtzq") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "kube-api-access-6xtzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:49:52.310078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.310021 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bd274e6-72e8-401c-958b-c7ef1f152273" (UID: "6bd274e6-72e8-401c-958b-c7ef1f152273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:49:52.371116 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.371089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:52.408837 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.408807 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtzq\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-kube-api-access-6xtzq\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.408837 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.408832 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-envoy\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.408999 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.408846 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-token\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.408999 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.408858 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd274e6-72e8-401c-958b-c7ef1f152273-istio-podinfo\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.408999 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.408868 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd274e6-72e8-401c-958b-c7ef1f152273-istiod-ca-cert\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:49:52.590777 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.590753 2559 scope.go:117] "RemoveContainer" containerID="960a993697b1dd17ecb8f774e3fc843b502b840f0f7f1e4f4d87115336cf78e5" Sep 15 21:49:52.597475 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.597442 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:49:52.607025 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:52.607004 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5dc07b-openshift-default-6bcb974fcd-r28ct"] Sep 15 21:49:53.172950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:53.172919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:54.107936 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:54.107900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:49:54.290126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:54.290098 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" path="/var/lib/kubelet/pods/6bd274e6-72e8-401c-958b-c7ef1f152273/volumes" Sep 15 21:49:54.989065 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:49:54.989032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:03.745514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:03.745472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:50:06.114172 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.114141 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:50:06.114569 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.114375 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" podUID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" containerName="maas-api" containerID="cri-o://928e3736db8a93b1e9746a61043649999d261bb9efc3209cf31ad37319063974" gracePeriod=30 Sep 15 21:50:06.150982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.150953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:06.251358 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.251324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:06.342325 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.342293 2559 generic.go:356] "Generic (PLEG): container finished" podID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" containerID="928e3736db8a93b1e9746a61043649999d261bb9efc3209cf31ad37319063974" exitCode=0 Sep 15 21:50:06.342489 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.342365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" event={"ID":"d8c6b381-a0ed-49b1-aaf2-cec011c62b82","Type":"ContainerDied","Data":"928e3736db8a93b1e9746a61043649999d261bb9efc3209cf31ad37319063974"} Sep 15 21:50:06.354036 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.354014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:06.370488 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.370422 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:50:06.427159 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427124 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8jrf\" (UniqueName: \"kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf\") pod \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " Sep 15 21:50:06.427324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427180 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs\") pod \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " Sep 15 21:50:06.427324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca\") pod \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " Sep 15 21:50:06.427446 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427362 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") pod \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\" (UID: \"d8c6b381-a0ed-49b1-aaf2-cec011c62b82\") " Sep 15 21:50:06.427621 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427601 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8c6b381-a0ed-49b1-aaf2-cec011c62b82" (UID: "d8c6b381-a0ed-49b1-aaf2-cec011c62b82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:50:06.427794 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.427779 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:06.429374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.429341 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8c6b381-a0ed-49b1-aaf2-cec011c62b82" (UID: "d8c6b381-a0ed-49b1-aaf2-cec011c62b82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:50:06.429509 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.429401 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8c6b381-a0ed-49b1-aaf2-cec011c62b82" (UID: "d8c6b381-a0ed-49b1-aaf2-cec011c62b82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:50:06.429509 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.429429 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf" (OuterVolumeSpecName: "kube-api-access-k8jrf") pod "d8c6b381-a0ed-49b1-aaf2-cec011c62b82" (UID: "d8c6b381-a0ed-49b1-aaf2-cec011c62b82"). InnerVolumeSpecName "kube-api-access-k8jrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:50:06.440254 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.440231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:06.528613 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.528576 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8jrf\" (UniqueName: \"kubernetes.io/projected/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-kube-api-access-k8jrf\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:06.528613 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.528614 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:06.528805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.528630 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6b381-a0ed-49b1-aaf2-cec011c62b82-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:06.813680 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.813602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:06.912992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:06.912963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:07.347967 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.347940 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" Sep 15 21:50:07.348390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.347976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv" event={"ID":"d8c6b381-a0ed-49b1-aaf2-cec011c62b82","Type":"ContainerDied","Data":"d526e48edee471db6bbf6800cb440793433ba19a0e2f2dc4fa069816643bbcd0"} Sep 15 21:50:07.348390 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.348030 2559 scope.go:117] "RemoveContainer" containerID="928e3736db8a93b1e9746a61043649999d261bb9efc3209cf31ad37319063974" Sep 15 21:50:07.372798 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.372773 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:50:07.379567 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.379542 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b2c5fe-754fdd679d-kw6lv"] Sep 15 21:50:07.779626 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:07.779542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:08.290600 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:08.290570 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" path="/var/lib/kubelet/pods/d8c6b381-a0ed-49b1-aaf2-cec011c62b82/volumes" Sep 15 21:50:08.788040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:08.788011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:09.585907 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:09.585876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:11.806882 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:11.806845 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:50:11.873314 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:11.873282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:17.063031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:17.062940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:50:26.825579 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:26.825531 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" podUID="8094d621-69f2-43c3-9baf-daa208ffe48b" containerName="main" containerID="cri-o://e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f" gracePeriod=45 Sep 15 21:50:27.077416 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.077353 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:50:27.207979 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.207947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.207987 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208018 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208094 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sh69\" (UniqueName: \"kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208217 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir\") pod \"8094d621-69f2-43c3-9baf-daa208ffe48b\" (UID: \"8094d621-69f2-43c3-9baf-daa208ffe48b\") " Sep 15 21:50:27.208514 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208217 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:50:27.208644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208359 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home" (OuterVolumeSpecName: "home") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:50:27.208644 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208388 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:50:27.208825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208708 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.208825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208728 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.208825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.208742 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.210740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.210712 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:50:27.210740 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.210720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69" (OuterVolumeSpecName: "kube-api-access-6sh69") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "kube-api-access-6sh69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:50:27.210926 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.210811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm" (OuterVolumeSpecName: "dshm") pod "8094d621-69f2-43c3-9baf-daa208ffe48b" (UID: "8094d621-69f2-43c3-9baf-daa208ffe48b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:50:27.309695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.309655 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094d621-69f2-43c3-9baf-daa208ffe48b-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.309695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.309689 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094d621-69f2-43c3-9baf-daa208ffe48b-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.309695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.309703 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sh69\" (UniqueName: \"kubernetes.io/projected/8094d621-69f2-43c3-9baf-daa208ffe48b-kube-api-access-6sh69\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:50:27.445154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.445059 2559 generic.go:356] "Generic (PLEG): container finished" podID="8094d621-69f2-43c3-9baf-daa208ffe48b" containerID="e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f" exitCode=0 Sep 15 21:50:27.445154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.445096 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" Sep 15 21:50:27.445154 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.445145 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" event={"ID":"8094d621-69f2-43c3-9baf-daa208ffe48b","Type":"ContainerDied","Data":"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f"} Sep 15 21:50:27.445441 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.445188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb" event={"ID":"8094d621-69f2-43c3-9baf-daa208ffe48b","Type":"ContainerDied","Data":"dad666dbca478b155248c2f6a0741312f32cd8a7fe3b8fd352558355e35f6414"} Sep 15 21:50:27.445441 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.445205 2559 scope.go:117] "RemoveContainer" containerID="e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f" Sep 15 21:50:27.454761 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.454746 2559 scope.go:117] "RemoveContainer" containerID="e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f" Sep 15 21:50:27.455016 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:50:27.455000 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f\": container with ID starting with e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f not found: ID does not exist" containerID="e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f" Sep 15 21:50:27.455065 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.455024 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f"} err="failed to get container status \"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f\": rpc error: code = NotFound desc = could not find container \"e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f\": container with ID starting with e95083ce39c8dcd26eeaf7d9674428dc6cc9845f05edc5821bb8d6776c07770f not found: ID does not exist" Sep 15 21:50:27.469015 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.468991 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:50:27.475299 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:27.475278 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b2c5fe/ipp-route-b2c5fe-kserve-8475db7c9f-fx4hb"] Sep 15 21:50:28.290191 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:28.290149 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8094d621-69f2-43c3-9baf-daa208ffe48b" path="/var/lib/kubelet/pods/8094d621-69f2-43c3-9baf-daa208ffe48b/volumes" Sep 15 21:50:47.986835 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:47.986798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:49.383083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:49.383049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:55.259323 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:55.259266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:55.359980 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:55.359945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:55.550093 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:55.550017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:56.772085 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:56.772029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:57.773388 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:57.773344 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:58.576704 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:58.576663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:50:59.586664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:50:59.586632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:06.569960 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:06.569927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:06.638816 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:06.638785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:06.747981 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:06.747957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:06.848363 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:06.848338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:07.159640 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:07.159544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:07.272324 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:07.272292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:08.317951 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:08.317918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:09.255397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:09.255364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:09.757825 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:09.757787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:51:13.153428 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:13.153389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:13.254262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:13.254230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:13.464927 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:13.464856 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:14.077211 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:14.077178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:15.132810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:15.132779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:15.882675 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:15.882643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:16.648180 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:16.648139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:17.106904 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:17.106869 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:24.448259 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.448203 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:51:24.448737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.448490 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" podUID="07856ddb-f228-4c4a-9826-73f0550a32b5" containerName="maas-api" containerID="cri-o://b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2" gracePeriod=30 Sep 15 21:51:24.491974 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.491930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:24.591364 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.590440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:24.692955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.692928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:24.695742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.695725 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:51:24.717386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.717318 2559 generic.go:356] "Generic (PLEG): container finished" podID="07856ddb-f228-4c4a-9826-73f0550a32b5" containerID="b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2" exitCode=0 Sep 15 21:51:24.717386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.717364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" event={"ID":"07856ddb-f228-4c4a-9826-73f0550a32b5","Type":"ContainerDied","Data":"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2"} Sep 15 21:51:24.717595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.717386 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" Sep 15 21:51:24.717595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.717420 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw" event={"ID":"07856ddb-f228-4c4a-9826-73f0550a32b5","Type":"ContainerDied","Data":"8220bac8b3620c1aa48e48415516d9b556614ee8d4d7abdb5cf2f56ee422a1d4"} Sep 15 21:51:24.717595 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.717471 2559 scope.go:117] "RemoveContainer" containerID="b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2" Sep 15 21:51:24.727935 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.727915 2559 scope.go:117] "RemoveContainer" containerID="b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2" Sep 15 21:51:24.728205 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:51:24.728182 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2\": container with ID starting with b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2 not found: ID does not exist" containerID="b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2" Sep 15 21:51:24.728257 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.728214 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2"} err="failed to get container status \"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2\": rpc error: code = NotFound desc = could not find container \"b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2\": container with ID starting with b38baa1c598434139b87b89d831257b77b68c8cbb4205be9dbd4d67bbde2f1d2 not found: ID does not exist" Sep 15 21:51:24.774666 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.774637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:24.803771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.803734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghcm2\" (UniqueName: \"kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2\") pod \"07856ddb-f228-4c4a-9826-73f0550a32b5\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " Sep 15 21:51:24.803889 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.803832 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs\") pod \"07856ddb-f228-4c4a-9826-73f0550a32b5\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " Sep 15 21:51:24.803889 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.803858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca\") pod \"07856ddb-f228-4c4a-9826-73f0550a32b5\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " Sep 15 21:51:24.804010 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.803921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") pod \"07856ddb-f228-4c4a-9826-73f0550a32b5\" (UID: \"07856ddb-f228-4c4a-9826-73f0550a32b5\") " Sep 15 21:51:24.804323 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.804291 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07856ddb-f228-4c4a-9826-73f0550a32b5" (UID: "07856ddb-f228-4c4a-9826-73f0550a32b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:24.806254 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.806225 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2" (OuterVolumeSpecName: "kube-api-access-ghcm2") pod "07856ddb-f228-4c4a-9826-73f0550a32b5" (UID: "07856ddb-f228-4c4a-9826-73f0550a32b5"). InnerVolumeSpecName "kube-api-access-ghcm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:24.806384 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.806364 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07856ddb-f228-4c4a-9826-73f0550a32b5" (UID: "07856ddb-f228-4c4a-9826-73f0550a32b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:51:24.806477 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.806387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07856ddb-f228-4c4a-9826-73f0550a32b5" (UID: "07856ddb-f228-4c4a-9826-73f0550a32b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:51:24.905022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.904987 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:24.905022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.905013 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07856ddb-f228-4c4a-9826-73f0550a32b5-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:24.905022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.905024 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07856ddb-f228-4c4a-9826-73f0550a32b5-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:24.905255 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:24.905033 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcm2\" (UniqueName: \"kubernetes.io/projected/07856ddb-f228-4c4a-9826-73f0550a32b5-kube-api-access-ghcm2\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:25.060205 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:25.060167 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:51:25.062796 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:25.062773 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59e243-6897dbf4d4-8m9xw"] Sep 15 21:51:25.156599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:25.156573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:25.259934 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:25.259899 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:26.001229 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:26.001197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:26.290203 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:26.290126 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07856ddb-f228-4c4a-9826-73f0550a32b5" path="/var/lib/kubelet/pods/07856ddb-f228-4c4a-9826-73f0550a32b5/volumes" Sep 15 21:51:27.522853 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:27.522820 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:51:31.169833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:31.169802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:31.266103 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:31.266072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:31.438501 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:31.438411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:32.721647 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:32.721618 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:33.582071 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:33.582034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:42.314677 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.314646 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:51:42.315185 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.314846 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" podUID="959b8124-8735-4f80-a2b5-545d1887e296" containerName="maas-api" containerID="cri-o://97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f" gracePeriod=30 Sep 15 21:51:42.347278 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.347256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:42.442667 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.442634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:42.542141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.542113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:42.553442 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.553423 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:51:42.642138 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.642115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:42.655198 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.655177 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca\") pod \"959b8124-8735-4f80-a2b5-545d1887e296\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " Sep 15 21:51:42.655298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.655205 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs\") pod \"959b8124-8735-4f80-a2b5-545d1887e296\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " Sep 15 21:51:42.655298 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.655264 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls\") pod \"959b8124-8735-4f80-a2b5-545d1887e296\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " Sep 15 21:51:42.655396 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.655312 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv86l\" (UniqueName: \"kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l\") pod \"959b8124-8735-4f80-a2b5-545d1887e296\" (UID: \"959b8124-8735-4f80-a2b5-545d1887e296\") " Sep 15 21:51:42.655616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.655581 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "959b8124-8735-4f80-a2b5-545d1887e296" (UID: "959b8124-8735-4f80-a2b5-545d1887e296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:51:42.657315 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.657288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "959b8124-8735-4f80-a2b5-545d1887e296" (UID: "959b8124-8735-4f80-a2b5-545d1887e296"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:51:42.657421 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.657355 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "959b8124-8735-4f80-a2b5-545d1887e296" (UID: "959b8124-8735-4f80-a2b5-545d1887e296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:51:42.657421 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.657368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l" (OuterVolumeSpecName: "kube-api-access-vv86l") pod "959b8124-8735-4f80-a2b5-545d1887e296" (UID: "959b8124-8735-4f80-a2b5-545d1887e296"). InnerVolumeSpecName "kube-api-access-vv86l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:51:42.756652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.756621 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959b8124-8735-4f80-a2b5-545d1887e296-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:42.756652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.756649 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:42.756652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.756658 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959b8124-8735-4f80-a2b5-545d1887e296-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:42.756913 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.756667 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv86l\" (UniqueName: \"kubernetes.io/projected/959b8124-8735-4f80-a2b5-545d1887e296-kube-api-access-vv86l\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:51:42.803712 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.803681 2559 generic.go:356] "Generic (PLEG): container finished" podID="959b8124-8735-4f80-a2b5-545d1887e296" containerID="97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f" exitCode=0 Sep 15 21:51:42.803887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.803713 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" event={"ID":"959b8124-8735-4f80-a2b5-545d1887e296","Type":"ContainerDied","Data":"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f"} Sep 15 21:51:42.803887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.803744 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" Sep 15 21:51:42.803887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.803805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m" event={"ID":"959b8124-8735-4f80-a2b5-545d1887e296","Type":"ContainerDied","Data":"c7dba8dcf2c740167f27343dc9131b94cfeb9fbe4b1dfd30abc19e3fee400b4a"} Sep 15 21:51:42.803887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.803840 2559 scope.go:117] "RemoveContainer" containerID="97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f" Sep 15 21:51:42.814096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.814046 2559 scope.go:117] "RemoveContainer" containerID="97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f" Sep 15 21:51:42.814317 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:51:42.814296 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f\": container with ID starting with 97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f not found: ID does not exist" containerID="97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f" Sep 15 21:51:42.814397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.814323 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f"} err="failed to get container status \"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f\": rpc error: code = NotFound desc = could not find container \"97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f\": container with ID starting with 97886c62bedd3149602ae28371222dd6f0e0d9f41e9cfd20094faf1e5510cf3f not found: ID does not exist" Sep 15 21:51:42.828878 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.828856 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:51:42.834223 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:42.834201 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-38a3a3-7fbcd7fd8b-hq49m"] Sep 15 21:51:43.038265 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:43.038173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:43.195523 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:43.195493 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:44.290016 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:44.289986 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959b8124-8735-4f80-a2b5-545d1887e296" path="/var/lib/kubelet/pods/959b8124-8735-4f80-a2b5-545d1887e296/volumes" Sep 15 21:51:45.146581 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:45.146503 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:51:49.554476 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:49.554431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:49.654985 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:49.654955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:49.949044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:49.949010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:51:54.529386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:51:54.529348 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:53:00.868829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:00.868798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:53:14.116955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:14.116920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:53:14.521829 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:14.521750 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:53:15.808209 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:15.808123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:53:39.923028 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:39.922993 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:53:47.719223 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:47.719188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:53:54.817652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:53:54.817615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:54:02.324050 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:02.324016 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:54:15.333262 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:15.333227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:54:15.387136 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:15.387106 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:54:17.034231 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:17.034201 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:54:26.614877 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:26.614841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:54:31.067598 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:31.067565 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:54:31.168152 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:31.168122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:54:31.801420 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:31.801388 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:54:31.801646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:31.801610 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" containerID="cri-o://dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2" gracePeriod=10 Sep 15 21:54:32.049788 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.049758 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:54:32.167628 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.167602 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bknkg\" (UniqueName: \"kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg\") pod \"484d69e4-1c76-4088-bca6-fedb65acaa23\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " Sep 15 21:54:32.167993 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.167697 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume\") pod \"484d69e4-1c76-4088-bca6-fedb65acaa23\" (UID: \"484d69e4-1c76-4088-bca6-fedb65acaa23\") " Sep 15 21:54:32.168078 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.168058 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "484d69e4-1c76-4088-bca6-fedb65acaa23" (UID: "484d69e4-1c76-4088-bca6-fedb65acaa23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:54:32.169588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.169566 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg" (OuterVolumeSpecName: "kube-api-access-bknkg") pod "484d69e4-1c76-4088-bca6-fedb65acaa23" (UID: "484d69e4-1c76-4088-bca6-fedb65acaa23"). InnerVolumeSpecName "kube-api-access-bknkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:54:32.269183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.269155 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/484d69e4-1c76-4088-bca6-fedb65acaa23-extensions-socket-volume\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:54:32.269183 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.269179 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bknkg\" (UniqueName: \"kubernetes.io/projected/484d69e4-1c76-4088-bca6-fedb65acaa23-kube-api-access-bknkg\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:54:32.601836 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.601800 2559 generic.go:356] "Generic (PLEG): container finished" podID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerID="dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2" exitCode=0 Sep 15 21:54:32.601997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.601850 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" Sep 15 21:54:32.601997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.601878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerDied","Data":"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2"} Sep 15 21:54:32.601997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.601931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289" event={"ID":"484d69e4-1c76-4088-bca6-fedb65acaa23","Type":"ContainerDied","Data":"c8da05744cf6233a4854d0a8030180bcde827d3141fa119fdac243374861a6d6"} Sep 15 21:54:32.601997 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.601952 2559 scope.go:117] "RemoveContainer" containerID="dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2" Sep 15 21:54:32.610335 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.610312 2559 scope.go:117] "RemoveContainer" containerID="2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314" Sep 15 21:54:32.617901 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.617878 2559 scope.go:117] "RemoveContainer" containerID="dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2" Sep 15 21:54:32.618262 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:54:32.618226 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2\": container with ID starting with dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2 not found: ID does not exist" containerID="dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2" Sep 15 21:54:32.618352 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.618260 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2"} err="failed to get container status \"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2\": rpc error: code = NotFound desc = could not find container \"dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2\": container with ID starting with dd9a7cdd4fa371275c00266a81f18d9354c368789d9a9e7bc6025393392253f2 not found: ID does not exist" Sep 15 21:54:32.618352 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.618284 2559 scope.go:117] "RemoveContainer" containerID="2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314" Sep 15 21:54:32.618585 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:54:32.618566 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314\": container with ID starting with 2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314 not found: ID does not exist" containerID="2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314" Sep 15 21:54:32.618635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.618590 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314"} err="failed to get container status \"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314\": rpc error: code = NotFound desc = could not find container \"2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314\": container with ID starting with 2d18b8bb2f356ef56c69cc2710f266a52b4aba976ec6f1daedf12b3db7ad5314 not found: ID does not exist" Sep 15 21:54:32.620120 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.620101 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:54:32.623880 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:32.623860 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n8289"] Sep 15 21:54:34.294182 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:54:34.294149 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" path="/var/lib/kubelet/pods/484d69e4-1c76-4088-bca6-fedb65acaa23/volumes" Sep 15 21:55:32.480176 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480143 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl"] Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480483 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480495 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480506 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480512 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480525 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480536 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480544 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480550 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480557 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07856ddb-f228-4c4a-9826-73f0550a32b5" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480562 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07856ddb-f228-4c4a-9826-73f0550a32b5" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480569 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959b8124-8735-4f80-a2b5-545d1887e296" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480574 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="959b8124-8735-4f80-a2b5-545d1887e296" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480581 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8094d621-69f2-43c3-9baf-daa208ffe48b" containerName="main" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480586 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8094d621-69f2-43c3-9baf-daa208ffe48b" containerName="main" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480650 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="959b8124-8735-4f80-a2b5-545d1887e296" containerName="maas-api" Sep 15 21:55:32.480650 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480659 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c6b381-a0ed-49b1-aaf2-cec011c62b82" containerName="maas-api" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480664 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480671 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8094d621-69f2-43c3-9baf-daa208ffe48b" containerName="main" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480678 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="484d69e4-1c76-4088-bca6-fedb65acaa23" containerName="manager" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480686 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480692 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="07856ddb-f228-4c4a-9826-73f0550a32b5" containerName="maas-api" Sep 15 21:55:32.481128 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.480696 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:55:32.483642 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.483626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.486652 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.486630 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-45z78\"" Sep 15 21:55:32.494918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.494897 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl"] Sep 15 21:55:32.522301 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.522277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2754b16-d7c4-452d-915b-1beea400f19d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.522436 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.522330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbvd\" (UniqueName: \"kubernetes.io/projected/a2754b16-d7c4-452d-915b-1beea400f19d-kube-api-access-8hbvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.623635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.623603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbvd\" (UniqueName: \"kubernetes.io/projected/a2754b16-d7c4-452d-915b-1beea400f19d-kube-api-access-8hbvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.623806 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.623688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2754b16-d7c4-452d-915b-1beea400f19d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.624007 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.623992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2754b16-d7c4-452d-915b-1beea400f19d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.632955 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.632930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbvd\" (UniqueName: \"kubernetes.io/projected/a2754b16-d7c4-452d-915b-1beea400f19d-kube-api-access-8hbvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nmmwl\" (UID: \"a2754b16-d7c4-452d-915b-1beea400f19d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:32.794555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:32.794470 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:33.072096 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:33.072022 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl"] Sep 15 21:55:33.075369 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:55:33.075340 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2754b16_d7c4_452d_915b_1beea400f19d.slice/crio-1b730e94cf53b0be4034e86d251eeed9a6c5dcb24cfb5410cbd2210785198d6f WatchSource:0}: Error finding container 1b730e94cf53b0be4034e86d251eeed9a6c5dcb24cfb5410cbd2210785198d6f: Status 404 returned error can't find the container with id 1b730e94cf53b0be4034e86d251eeed9a6c5dcb24cfb5410cbd2210785198d6f Sep 15 21:55:33.881963 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:33.881927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" event={"ID":"a2754b16-d7c4-452d-915b-1beea400f19d","Type":"ContainerStarted","Data":"f610ef19e0348b424b74ab206a7812bb1e80cf374e6c26954daec60b3b57c7f4"} Sep 15 21:55:33.882433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:33.881983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" event={"ID":"a2754b16-d7c4-452d-915b-1beea400f19d","Type":"ContainerStarted","Data":"1b730e94cf53b0be4034e86d251eeed9a6c5dcb24cfb5410cbd2210785198d6f"} Sep 15 21:55:33.882433 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:33.882055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:33.904273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:33.904222 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" podStartSLOduration=1.904209061 podStartE2EDuration="1.904209061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:55:33.901297617 +0000 UTC m=+2382.125920633" watchObservedRunningTime="2026-09-15 21:55:33.904209061 +0000 UTC m=+2382.128832064" Sep 15 21:55:44.888683 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:44.888647 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nmmwl" Sep 15 21:55:51.911468 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:51.911433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:55:58.916666 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:55:58.916624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:56:06.811141 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:06.811109 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:56:27.215121 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.215041 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:56:27.215627 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.215606 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:56:27.215679 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.215630 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd274e6-72e8-401c-958b-c7ef1f152273" containerName="istio-proxy" Sep 15 21:56:27.219085 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.219065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.222296 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.222275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f2776979-kserve-self-signed-certs\"" Sep 15 21:56:27.227919 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.227898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:56:27.278144 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.278284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278149 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.278284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlb8\" (UniqueName: \"kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.278410 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.278410 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.278410 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.278389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.281032 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.281002 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:56:27.378943 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.378919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.378951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.378999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlb8\" (UniqueName: \"kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379083 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379374 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379586 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.379717 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.379663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.381354 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.381327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.382151 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.382108 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.387490 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.387470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlb8\" (UniqueName: \"kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8\") pod \"e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.530665 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.530604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:27.804265 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:27.804240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:56:27.806489 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:56:27.806445 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23c3bb0f_1cbb_4726_96f7_ca4d442bf999.slice/crio-4ce60daeb0f494fbd5fe142a584f255676184a875f0b32fc8bad1a964ec12d94 WatchSource:0}: Error finding container 4ce60daeb0f494fbd5fe142a584f255676184a875f0b32fc8bad1a964ec12d94: Status 404 returned error can't find the container with id 4ce60daeb0f494fbd5fe142a584f255676184a875f0b32fc8bad1a964ec12d94 Sep 15 21:56:28.140292 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:28.140255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" event={"ID":"23c3bb0f-1cbb-4726-96f7-ca4d442bf999","Type":"ContainerStarted","Data":"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb"} Sep 15 21:56:28.140450 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:28.140299 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" event={"ID":"23c3bb0f-1cbb-4726-96f7-ca4d442bf999","Type":"ContainerStarted","Data":"4ce60daeb0f494fbd5fe142a584f255676184a875f0b32fc8bad1a964ec12d94"} Sep 15 21:56:28.161505 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:28.161403 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" podStartSLOduration=1.16138817 podStartE2EDuration="1.16138817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:56:28.160626074 +0000 UTC m=+2436.385249081" watchObservedRunningTime="2026-09-15 21:56:28.16138817 +0000 UTC m=+2436.386011498" Sep 15 21:56:29.822232 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:29.822195 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:56:36.531184 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:36.531152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:56:37.531699 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:37.531665 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:37.532149 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:37.531717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:37.544031 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:37.544003 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:38.199388 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:38.199362 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:56:39.516519 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:56:39.516487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:09.998633 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:09.998590 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:57:10.026114 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:10.026077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:57:10.829055 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:10.829023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:57:13.130695 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:13.130658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:22.542636 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:22.542605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:25.018837 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.018797 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" podUID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" containerName="main" containerID="cri-o://856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb" gracePeriod=45 Sep 15 21:57:25.260284 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.260258 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:57:25.406426 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.406392 2559 generic.go:356] "Generic (PLEG): container finished" podID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" containerID="856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb" exitCode=0 Sep 15 21:57:25.406616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.406448 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" Sep 15 21:57:25.406616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.406484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" event={"ID":"23c3bb0f-1cbb-4726-96f7-ca4d442bf999","Type":"ContainerDied","Data":"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb"} Sep 15 21:57:25.406616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.406524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv" event={"ID":"23c3bb0f-1cbb-4726-96f7-ca4d442bf999","Type":"ContainerDied","Data":"4ce60daeb0f494fbd5fe142a584f255676184a875f0b32fc8bad1a964ec12d94"} Sep 15 21:57:25.406616 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.406540 2559 scope.go:117] "RemoveContainer" containerID="856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb" Sep 15 21:57:25.414672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.414656 2559 scope.go:117] "RemoveContainer" containerID="856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb" Sep 15 21:57:25.414947 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:57:25.414929 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb\": container with ID starting with 856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb not found: ID does not exist" containerID="856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb" Sep 15 21:57:25.415003 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.414955 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb"} err="failed to get container status \"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb\": rpc error: code = NotFound desc = could not find container \"856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb\": container with ID starting with 856d724fdd2aab3403d74d643a64984241af7030b283b4ab4e8ce10475ded3eb not found: ID does not exist" Sep 15 21:57:25.453268 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453322 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453351 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453418 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453528 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dlb8\" (UniqueName: \"kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8\") pod \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\" (UID: \"23c3bb0f-1cbb-4726-96f7-ca4d442bf999\") " Sep 15 21:57:25.453622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home" (OuterVolumeSpecName: "home") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:25.453773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453650 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache" (OuterVolumeSpecName: "model-cache") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:25.453773 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.453687 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:25.454024 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.454004 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-home\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.454110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.454029 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-model-cache\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.454110 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.454044 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tmp-dir\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.455753 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.455729 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm" (OuterVolumeSpecName: "dshm") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:57:25.455753 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.455732 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:57:25.455887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.455804 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8" (OuterVolumeSpecName: "kube-api-access-6dlb8") pod "23c3bb0f-1cbb-4726-96f7-ca4d442bf999" (UID: "23c3bb0f-1cbb-4726-96f7-ca4d442bf999"). InnerVolumeSpecName "kube-api-access-6dlb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:57:25.554896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.554866 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-tls-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.554896 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.554893 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-dshm\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.555068 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.554903 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlb8\" (UniqueName: \"kubernetes.io/projected/23c3bb0f-1cbb-4726-96f7-ca4d442bf999-kube-api-access-6dlb8\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:25.733977 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.733948 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:57:25.736916 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:25.736892 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f2776979-kserve-dc95f4795-hfxcv"] Sep 15 21:57:26.291716 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:26.291682 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" path="/var/lib/kubelet/pods/23c3bb0f-1cbb-4726-96f7-ca4d442bf999/volumes" Sep 15 21:57:29.057526 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.057480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:57:29.439763 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.439729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:29.670566 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.670533 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:29.670874 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.670862 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" containerName="main" Sep 15 21:57:29.670918 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.670875 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" containerName="main" Sep 15 21:57:29.670956 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.670947 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c3bb0f-1cbb-4726-96f7-ca4d442bf999" containerName="main" Sep 15 21:57:29.675389 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.675371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.680612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.680584 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 21:57:29.680612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.680585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 21:57:29.680810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.680645 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 21:57:29.680810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.680657 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 21:57:29.680810 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.680800 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sgjsd\"" Sep 15 21:57:29.686273 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.686239 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:29.788742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.788661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.788742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.788700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.788923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.788828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.788923 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.788878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vjd\" (UniqueName: \"kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.889731 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.889701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.889866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.889764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vjd\" (UniqueName: \"kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.889866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.889820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.889866 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.889845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.890382 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.890361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.892409 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.892379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.892555 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.892537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.899814 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.899782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vjd\" (UniqueName: \"kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd\") pod \"maas-api-97c9c8fdc-hj5g9\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:29.985601 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:29.985571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:30.262560 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:30.262138 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:30.264984 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:57:30.264931 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca34441_cd6f_4280_92d7_2d3c649dda43.slice/crio-4e1e778891d8ae0aca9edd9141461b47068138e3ccf8a1dfcda6955debe71719 WatchSource:0}: Error finding container 4e1e778891d8ae0aca9edd9141461b47068138e3ccf8a1dfcda6955debe71719: Status 404 returned error can't find the container with id 4e1e778891d8ae0aca9edd9141461b47068138e3ccf8a1dfcda6955debe71719 Sep 15 21:57:30.266906 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:30.266888 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:57:30.430833 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:30.430801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" event={"ID":"aca34441-cd6f-4280-92d7-2d3c649dda43","Type":"ContainerStarted","Data":"4e1e778891d8ae0aca9edd9141461b47068138e3ccf8a1dfcda6955debe71719"} Sep 15 21:57:31.436397 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:31.436362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" event={"ID":"aca34441-cd6f-4280-92d7-2d3c649dda43","Type":"ContainerStarted","Data":"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e"} Sep 15 21:57:31.436821 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:31.436497 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:31.468816 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:31.468756 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" podStartSLOduration=1.9221099050000001 podStartE2EDuration="2.468734514s" totalImagesPullingTime="546.624609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:31.468380484 +0000 UTC m=+2499.693003487" watchObservedRunningTime="2026-09-15 21:57:31.468734514 +0000 UTC m=+2499.693357519" Sep 15 21:57:37.447805 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.447776 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:37.468950 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.468918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:37.642308 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.642278 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm"] Sep 15 21:57:37.645759 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.645740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.670012 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.669984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm"] Sep 15 21:57:37.745362 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.745285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-maas-api-tls\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.745362 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.745320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd4591-fe2f-49e5-ad17-713a7041f848-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.745599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.745492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-metrics-certs\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.745599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.745536 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrhv\" (UniqueName: \"kubernetes.io/projected/76bd4591-fe2f-49e5-ad17-713a7041f848-kube-api-access-nkrhv\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.846840 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.846809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-metrics-certs\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.846982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.846847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrhv\" (UniqueName: \"kubernetes.io/projected/76bd4591-fe2f-49e5-ad17-713a7041f848-kube-api-access-nkrhv\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.846982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.846900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-maas-api-tls\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.846982 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.846925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd4591-fe2f-49e5-ad17-713a7041f848-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.847651 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.847627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd4591-fe2f-49e5-ad17-713a7041f848-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.849317 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.849291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-maas-api-tls\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.849412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.849333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bd4591-fe2f-49e5-ad17-713a7041f848-metrics-certs\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.869044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.869016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrhv\" (UniqueName: \"kubernetes.io/projected/76bd4591-fe2f-49e5-ad17-713a7041f848-kube-api-access-nkrhv\") pod \"maas-api-85c7dfbd4c-726dm\" (UID: \"76bd4591-fe2f-49e5-ad17-713a7041f848\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:37.956412 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:37.956386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:38.243069 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:38.243039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm"] Sep 15 21:57:38.246520 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:57:38.246489 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bd4591_fe2f_49e5_ad17_713a7041f848.slice/crio-0aa3ff69495c3216cf2b6ef4a062277a4034d42f0a4798d48ffdf6974f8fb58c WatchSource:0}: Error finding container 0aa3ff69495c3216cf2b6ef4a062277a4034d42f0a4798d48ffdf6974f8fb58c: Status 404 returned error can't find the container with id 0aa3ff69495c3216cf2b6ef4a062277a4034d42f0a4798d48ffdf6974f8fb58c Sep 15 21:57:38.473386 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:38.473344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" event={"ID":"76bd4591-fe2f-49e5-ad17-713a7041f848","Type":"ContainerStarted","Data":"0aa3ff69495c3216cf2b6ef4a062277a4034d42f0a4798d48ffdf6974f8fb58c"} Sep 15 21:57:39.479672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:39.479635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" event={"ID":"76bd4591-fe2f-49e5-ad17-713a7041f848","Type":"ContainerStarted","Data":"4437da5360c70fa537917f793f6265991be5c5655da4d9066e0a7934e9373827"} Sep 15 21:57:39.480044 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:39.479724 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:44.348992 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:44.348941 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" podStartSLOduration=7.054859714 podStartE2EDuration="7.348925007s" totalImagesPullingTime="294.065293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:39.530997506 +0000 UTC m=+2507.755620510" watchObservedRunningTime="2026-09-15 21:57:44.348925007 +0000 UTC m=+2512.573548014" Sep 15 21:57:44.349465 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:44.349079 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:45.490009 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.489937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-726dm" Sep 15 21:57:45.602099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.602064 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:45.602294 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.602271 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" podUID="aca34441-cd6f-4280-92d7-2d3c649dda43" containerName="maas-api" containerID="cri-o://7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e" gracePeriod=30 Sep 15 21:57:45.844274 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.844252 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:45.921428 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.921399 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vjd\" (UniqueName: \"kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd\") pod \"aca34441-cd6f-4280-92d7-2d3c649dda43\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " Sep 15 21:57:45.921632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.921475 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca\") pod \"aca34441-cd6f-4280-92d7-2d3c649dda43\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " Sep 15 21:57:45.921632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.921496 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs\") pod \"aca34441-cd6f-4280-92d7-2d3c649dda43\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " Sep 15 21:57:45.921632 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.921572 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls\") pod \"aca34441-cd6f-4280-92d7-2d3c649dda43\" (UID: \"aca34441-cd6f-4280-92d7-2d3c649dda43\") " Sep 15 21:57:45.921885 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.921859 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aca34441-cd6f-4280-92d7-2d3c649dda43" (UID: "aca34441-cd6f-4280-92d7-2d3c649dda43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:57:45.923549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.923519 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd" (OuterVolumeSpecName: "kube-api-access-m8vjd") pod "aca34441-cd6f-4280-92d7-2d3c649dda43" (UID: "aca34441-cd6f-4280-92d7-2d3c649dda43"). InnerVolumeSpecName "kube-api-access-m8vjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:57:45.923549 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.923534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aca34441-cd6f-4280-92d7-2d3c649dda43" (UID: "aca34441-cd6f-4280-92d7-2d3c649dda43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:57:45.923690 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:45.923548 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aca34441-cd6f-4280-92d7-2d3c649dda43" (UID: "aca34441-cd6f-4280-92d7-2d3c649dda43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:57:46.022886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.022798 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-maas-api-tls\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:46.022886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.022830 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vjd\" (UniqueName: \"kubernetes.io/projected/aca34441-cd6f-4280-92d7-2d3c649dda43-kube-api-access-m8vjd\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:46.022886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.022851 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca34441-cd6f-4280-92d7-2d3c649dda43-openshift-service-ca\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:46.022886 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.022860 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca34441-cd6f-4280-92d7-2d3c649dda43-metrics-certs\") on node \"ip-10-0-129-100.ec2.internal\" DevicePath \"\"" Sep 15 21:57:46.513498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.513433 2559 generic.go:356] "Generic (PLEG): container finished" podID="aca34441-cd6f-4280-92d7-2d3c649dda43" containerID="7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e" exitCode=0 Sep 15 21:57:46.513498 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.513498 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" Sep 15 21:57:46.513988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.513525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" event={"ID":"aca34441-cd6f-4280-92d7-2d3c649dda43","Type":"ContainerDied","Data":"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e"} Sep 15 21:57:46.513988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.513580 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9" event={"ID":"aca34441-cd6f-4280-92d7-2d3c649dda43","Type":"ContainerDied","Data":"4e1e778891d8ae0aca9edd9141461b47068138e3ccf8a1dfcda6955debe71719"} Sep 15 21:57:46.513988 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.513606 2559 scope.go:117] "RemoveContainer" containerID="7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e" Sep 15 21:57:46.522099 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.522082 2559 scope.go:117] "RemoveContainer" containerID="7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e" Sep 15 21:57:46.522368 ip-10-0-129-100 kubenswrapper[2559]: E0915 21:57:46.522349 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e\": container with ID starting with 7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e not found: ID does not exist" containerID="7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e" Sep 15 21:57:46.522443 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.522377 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e"} err="failed to get container status \"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e\": rpc error: code = NotFound desc = could not find container \"7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e\": container with ID starting with 7f2c236287127a0681238a3aed7ab3b606eb78ab16330880b7b71fc69e24a36e not found: ID does not exist" Sep 15 21:57:46.534349 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.534324 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:46.536860 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:46.536840 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-hj5g9"] Sep 15 21:57:48.291077 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:48.291047 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca34441-cd6f-4280-92d7-2d3c649dda43" path="/var/lib/kubelet/pods/aca34441-cd6f-4280-92d7-2d3c649dda43/volumes" Sep 15 21:57:50.234945 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:50.234911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:57:57.150467 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:57:57.150428 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gbtkd"] Sep 15 21:58:00.299622 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:00.299583 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:58:05.691393 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:05.691362 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-547vf_94f4cffc-2549-4841-9c76-5925407909f1/manager/0.log" Sep 15 21:58:05.996276 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:05.996193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jnx9r_f049b212-f487-4f01-858d-3ff00bb80425/server/0.log" Sep 15 21:58:06.126190 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:06.126161 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-srhr2_e258e6cf-526f-46fa-9a5f-c7ef5ae094ac/manager/0.log" Sep 15 21:58:06.554986 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:06.554955 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-p6g7m_10f8ab2b-7f47-4b12-8b13-648b4e20820a/copy-manifests/0.log" Sep 15 21:58:06.570071 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:06.570045 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-p6g7m_10f8ab2b-7f47-4b12-8b13-648b4e20820a/manager/0.log" Sep 15 21:58:07.047808 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:07.047777 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85c7dfbd4c-726dm_76bd4591-fe2f-49e5-ad17-713a7041f848/maas-api/0.log" Sep 15 21:58:07.156692 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:07.156663 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825130-c94bj_0fe65aed-2ad7-401f-b8f0-98fbf518fc89/cleanup/0.log" Sep 15 21:58:08.444771 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:08.444743 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8bg79_f6e43b9c-f6ab-4b0d-a466-b5bdef85714f/manager/0.log" Sep 15 21:58:08.693561 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:08.693532 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nmmwl_a2754b16-d7c4-452d-915b-1beea400f19d/manager/0.log" Sep 15 21:58:08.853021 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:08.852993 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gbtkd_d40035ef-9cb5-46dd-9167-ef5113a68871/limitador/0.log" Sep 15 21:58:09.319588 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:09.319509 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx_29c95cc0-5431-4388-81b1-b30d357e0873/istio-proxy/0.log" Sep 15 21:58:09.817552 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:09.817521 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/1.log" Sep 15 21:58:10.810887 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:10.810854 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-kpvpp_e7bc4a20-3914-4fa3-893c-161a4bb87192/main/0.log" Sep 15 21:58:11.180594 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:11.180558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f7hcs_5c45d377-c596-4aee-9fcf-31014767d8a8/main/0.log" Sep 15 21:58:11.301373 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:11.301339 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-s7xfj_ce248c16-7ecd-41ec-b506-42724441778e/main/0.log" Sep 15 21:58:17.093804 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:17.093760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:58:18.604741 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:18.604701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:58:19.726612 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:19.726582 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" limit=1024 Sep 15 21:58:20.866424 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:20.866394 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nckfc_3c426a60-e8a8-4486-88f5-af838b4007c0/global-pull-secret-syncer/0.log" Sep 15 21:58:21.127609 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:21.127530 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-45z4b_44e40931-3727-41f0-994c-e8adf49cc5ae/konnectivity-agent/0.log" Sep 15 21:58:21.329672 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:21.329644 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-100.ec2.internal_39d7ad530ef8927e7726fb19029c9345/haproxy/0.log" Sep 15 21:58:25.165073 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:25.164997 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8bg79_f6e43b9c-f6ab-4b0d-a466-b5bdef85714f/manager/0.log" Sep 15 21:58:25.370238 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:25.370204 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-nmmwl_a2754b16-d7c4-452d-915b-1beea400f19d/manager/0.log" Sep 15 21:58:25.485377 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:25.485291 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gbtkd_d40035ef-9cb5-46dd-9167-ef5113a68871/limitador/0.log" Sep 15 21:58:28.235572 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:28.235538 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf7fm_da79fc27-ce03-4f6e-bec1-9562778075f1/node-exporter/0.log" Sep 15 21:58:28.287823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:28.287794 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf7fm_da79fc27-ce03-4f6e-bec1-9562778075f1/kube-rbac-proxy/0.log" Sep 15 21:58:28.356664 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:28.356639 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf7fm_da79fc27-ce03-4f6e-bec1-9562778075f1/init-textfile/0.log" Sep 15 21:58:30.556686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.556642 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg"] Sep 15 21:58:30.557186 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.557166 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca34441-cd6f-4280-92d7-2d3c649dda43" containerName="maas-api" Sep 15 21:58:30.557266 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.557189 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca34441-cd6f-4280-92d7-2d3c649dda43" containerName="maas-api" Sep 15 21:58:30.557319 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.557281 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca34441-cd6f-4280-92d7-2d3c649dda43" containerName="maas-api" Sep 15 21:58:30.560167 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.560147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.565635 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.565615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"kube-root-ca.crt\"" Sep 15 21:58:30.565737 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.565633 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"openshift-service-ca.crt\"" Sep 15 21:58:30.566874 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.566854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7p98\"/\"default-dockercfg-87jck\"" Sep 15 21:58:30.585111 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.585084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg"] Sep 15 21:58:30.595540 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.595517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-lib-modules\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.595646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.595550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-proc\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.595646 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.595581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzq9\" (UniqueName: \"kubernetes.io/projected/3369853a-c92f-4e09-9c70-dbb668197293-kube-api-access-4jzq9\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.595760 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.595737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-podres\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.595817 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.595784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-sys\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.696703 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-proc\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.696868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzq9\" (UniqueName: \"kubernetes.io/projected/3369853a-c92f-4e09-9c70-dbb668197293-kube-api-access-4jzq9\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.696868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-proc\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.696868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-podres\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.696868 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-sys\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.697062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-lib-modules\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.697062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-sys\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.697062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-podres\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.697062 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.696991 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3369853a-c92f-4e09-9c70-dbb668197293-lib-modules\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.716092 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.716071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzq9\" (UniqueName: \"kubernetes.io/projected/3369853a-c92f-4e09-9c70-dbb668197293-kube-api-access-4jzq9\") pod \"perf-node-gather-daemonset-fc5rg\" (UID: \"3369853a-c92f-4e09-9c70-dbb668197293\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:30.870065 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:30.870042 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:31.153423 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.153384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg"] Sep 15 21:58:31.156003 ip-10-0-129-100 kubenswrapper[2559]: W0915 21:58:31.155977 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3369853a_c92f_4e09_9c70_dbb668197293.slice/crio-48aec8659d92bd5ae9f58afad74b69d004ffb698397a2448ed40bfa3e1b4e7cc WatchSource:0}: Error finding container 48aec8659d92bd5ae9f58afad74b69d004ffb698397a2448ed40bfa3e1b4e7cc: Status 404 returned error can't find the container with id 48aec8659d92bd5ae9f58afad74b69d004ffb698397a2448ed40bfa3e1b4e7cc Sep 15 21:58:31.378742 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.378708 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-jzcnx_e656b5c0-cb8d-47bd-ab1a-c3d10cc3c9ee/networking-console-plugin/0.log" Sep 15 21:58:31.716357 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.716321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" event={"ID":"3369853a-c92f-4e09-9c70-dbb668197293","Type":"ContainerStarted","Data":"761643878c28839c715ac1136374cde245ee525ab8829351162876549aca8768"} Sep 15 21:58:31.716788 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.716378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" event={"ID":"3369853a-c92f-4e09-9c70-dbb668197293","Type":"ContainerStarted","Data":"48aec8659d92bd5ae9f58afad74b69d004ffb698397a2448ed40bfa3e1b4e7cc"} Sep 15 21:58:31.716788 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.716404 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:31.760478 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:31.760417 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" podStartSLOduration=1.760403729 podStartE2EDuration="1.760403729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:31.759261819 +0000 UTC m=+2559.983884822" watchObservedRunningTime="2026-09-15 21:58:31.760403729 +0000 UTC m=+2559.985026732" Sep 15 21:58:34.775406 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:34.775376 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zdkrl_178eccc4-c353-40e4-aec0-c2aa478cb2fe/dns/0.log" Sep 15 21:58:34.798101 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:34.798071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zdkrl_178eccc4-c353-40e4-aec0-c2aa478cb2fe/kube-rbac-proxy/0.log" Sep 15 21:58:34.854130 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:34.854111 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss58r_947e6bd7-0aef-424b-8c05-ad8ac9aa28e7/dns-node-resolver/0.log" Sep 15 21:58:35.498509 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:35.498480 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzm7t_b4b0234d-2913-4587-b2de-6e9d61fa7293/node-ca/0.log" Sep 15 21:58:36.579849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:36.579824 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx_29c95cc0-5431-4388-81b1-b30d357e0873/istio-proxy/0.log" Sep 15 21:58:37.097532 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:37.097502 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/0.log" Sep 15 21:58:37.102413 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:37.102390 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/1.log" Sep 15 21:58:37.730344 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:37.730314 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-fc5rg" Sep 15 21:58:38.194856 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:38.194825 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mdqkl_a541b194-706b-4755-a04d-248fd6007a91/serve-healthcheck-canary/0.log" Sep 15 21:58:38.758159 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:38.758130 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gscvr_95f0f7c0-c210-4802-b1a7-972d5ba25315/kube-rbac-proxy/0.log" Sep 15 21:58:38.795920 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:38.795896 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gscvr_95f0f7c0-c210-4802-b1a7-972d5ba25315/exporter/0.log" Sep 15 21:58:38.844022 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:38.843996 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gscvr_95f0f7c0-c210-4802-b1a7-972d5ba25315/extractor/0.log" Sep 15 21:58:41.567578 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:41.567550 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx_29c95cc0-5431-4388-81b1-b30d357e0873/istio-proxy/0.log" Sep 15 21:58:42.100359 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.100326 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/0.log" Sep 15 21:58:42.106849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.106823 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/1.log" Sep 15 21:58:42.227153 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.227041 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-547vf_94f4cffc-2549-4841-9c76-5925407909f1/manager/0.log" Sep 15 21:58:42.317874 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.317844 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jnx9r_f049b212-f487-4f01-858d-3ff00bb80425/server/0.log" Sep 15 21:58:42.454686 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.454648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-srhr2_e258e6cf-526f-46fa-9a5f-c7ef5ae094ac/manager/0.log" Sep 15 21:58:42.647938 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.647901 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-p6g7m_10f8ab2b-7f47-4b12-8b13-648b4e20820a/manager/0.log" Sep 15 21:58:42.670599 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:42.670574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-p6g7m_10f8ab2b-7f47-4b12-8b13-648b4e20820a/copy-manifests/0.log" Sep 15 21:58:43.812823 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:43.812793 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hmwzx_29c95cc0-5431-4388-81b1-b30d357e0873/istio-proxy/0.log" Sep 15 21:58:44.095855 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:44.095817 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-dccb565dd-dmsdl_8a3a63fe-9659-434e-8863-ce99627131c2/manager/0.log" Sep 15 21:58:44.318849 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:44.318799 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/0.log" Sep 15 21:58:44.323723 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:44.323703 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-574ps_05505b04-0c94-467f-9e72-bc3af758b0c2/istio-proxy/1.log" Sep 15 21:58:49.097126 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:49.097099 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bs6k6_d53b978a-e206-4f11-bb71-1ddae2e7083e/migrator/0.log" Sep 15 21:58:49.120300 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:49.120273 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bs6k6_d53b978a-e206-4f11-bb71-1ddae2e7083e/graceful-termination/0.log" Sep 15 21:58:49.553040 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:49.552927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-pxqx9_d28eda98-9f3c-4c18-87ba-b0d65433ca10/kube-storage-version-migrator-operator/1.log" Sep 15 21:58:49.554739 ip-10-0-129-100 kubenswrapper[2559]: I0915 21:58:49.554693 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-pxqx9_d28eda98-9f3c-4c18-87ba-b0d65433ca10/kube-storage-version-migrator-operator/0.log"