Sep 18 22:00:14.229102 ip-10-0-133-172 systemd[1]: Starting Kubernetes Kubelet... Sep 18 22:00:14.638686 ip-10-0-133-172 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 22:00:14.638686 ip-10-0-133-172 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 22:00:14.638686 ip-10-0-133-172 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 22:00:14.638686 ip-10-0-133-172 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 22:00:14.639318 ip-10-0-133-172 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 22:00:14.639858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.639766 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 22:00:14.644198 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644182 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.644198 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644197 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644201 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644204 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644207 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644210 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644213 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644216 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644219 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644221 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644224 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644226 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644229 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644234 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644237 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644240 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644243 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644247 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644251 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644255 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.644271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644259 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644262 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644266 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644269 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644272 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644275 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644282 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644286 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644289 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644292 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644295 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644297 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644300 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644303 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644305 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644308 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644310 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644313 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644315 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644318 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.644709 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644320 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644323 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644325 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644328 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644330 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644332 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644335 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644337 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644340 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644342 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644345 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644347 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644349 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644352 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644356 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644358 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644361 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644364 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644367 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644369 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.645211 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644373 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644375 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644377 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644380 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644382 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644385 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644387 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644390 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644392 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644394 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644397 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644400 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644402 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644405 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644407 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644410 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644413 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644415 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644418 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644420 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.645698 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644423 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644425 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644428 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644431 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644433 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644436 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644439 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644442 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644444 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644447 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644449 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644452 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644454 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644457 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644459 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644462 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644464 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644467 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.644469 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645137 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.646188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645145 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645150 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645157 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645163 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645168 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645172 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645175 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645177 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645180 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645183 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645185 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645187 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645190 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645197 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645202 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645205 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645208 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645210 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645214 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.646662 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645217 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645220 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645223 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645225 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645228 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645230 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645233 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645236 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645238 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645241 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645243 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645246 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645248 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645251 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645253 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645255 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645258 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645261 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645263 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645266 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.647159 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645269 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645271 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645274 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645276 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645279 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645281 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645285 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645287 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645290 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645292 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645295 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645298 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645301 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645304 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645306 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645309 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645311 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645314 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645316 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645318 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.647642 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645320 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645323 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645325 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645328 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645331 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645333 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645335 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645338 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645341 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645344 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645346 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645348 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645351 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645353 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645356 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645360 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645363 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645365 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645368 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645370 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.648181 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645373 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645375 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645378 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645381 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645383 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645386 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645388 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645390 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645393 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645396 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645398 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645400 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645403 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645405 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645408 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645410 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645414 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645417 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645420 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.648756 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645799 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645805 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645809 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645813 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645817 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645820 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645823 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645826 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645828 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645831 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645833 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645836 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645838 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645841 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645844 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645846 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645849 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645852 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645854 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645857 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.649271 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645860 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645862 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645865 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645867 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645870 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645873 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645875 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645878 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645880 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645883 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645885 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645888 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645890 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645893 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645896 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645898 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645901 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645903 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645905 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645908 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.649813 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645910 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645913 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645915 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645918 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645920 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645923 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645926 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645928 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645931 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645934 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645936 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645939 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645941 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645944 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645946 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645949 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645952 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645954 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645957 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.650291 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645959 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645962 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645964 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645967 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645969 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645972 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645975 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645977 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645979 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645982 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645984 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645987 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645991 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645994 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.645997 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646000 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646002 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646005 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646008 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.650748 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646011 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646014 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646019 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646022 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646025 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646028 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646030 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646033 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646036 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646038 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646040 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646043 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646045 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646048 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646051 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646054 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646057 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646059 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646062 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646065 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.651251 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646067 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646144 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646183 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646190 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646195 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646199 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646202 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646206 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646209 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646212 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646215 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646219 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646222 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646225 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646230 2577 flags.go:64] FLAG: --cgroup-root="" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646233 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646235 2577 flags.go:64] FLAG: --client-ca-file="" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646238 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646241 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646245 2577 flags.go:64] FLAG: --cluster-domain="" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646247 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646250 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646253 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646256 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 22:00:14.651732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646260 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646263 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646266 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646269 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646272 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646276 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646279 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646282 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646285 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646289 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646291 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646294 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646297 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646300 2577 flags.go:64] FLAG: --enable-server="true" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646302 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646307 2577 flags.go:64] FLAG: --event-burst="100" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646310 2577 flags.go:64] FLAG: --event-qps="50" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646313 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646315 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646318 2577 flags.go:64] FLAG: --eviction-hard="" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646322 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646324 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646329 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646341 2577 flags.go:64] FLAG: --eviction-soft="" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646344 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 22:00:14.652381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646347 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646350 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646353 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646355 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646358 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646361 2577 flags.go:64] FLAG: --feature-gates="" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646365 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646368 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646370 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646373 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646377 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646379 2577 flags.go:64] FLAG: --help="false" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646383 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646386 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646389 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646392 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646395 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646399 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646401 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646404 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646407 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646410 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646412 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646415 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 22:00:14.653041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646418 2577 flags.go:64] FLAG: --kube-reserved="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646421 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646424 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646427 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646429 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646433 2577 flags.go:64] FLAG: --lock-file="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646436 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646445 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646448 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646453 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646456 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646458 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646461 2577 flags.go:64] FLAG: --logging-format="text" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646464 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646467 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646470 2577 flags.go:64] FLAG: --manifest-url="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646473 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646477 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646480 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646483 2577 flags.go:64] FLAG: --max-pods="110" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646487 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646490 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646493 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646495 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646498 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 22:00:14.653621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646501 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646504 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646510 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646513 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646516 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646519 2577 flags.go:64] FLAG: --pod-cidr="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646521 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646526 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646529 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646532 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646534 2577 flags.go:64] FLAG: --port="10250" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646537 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646540 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05f5affa046230b5f" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646544 2577 flags.go:64] FLAG: --qos-reserved="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646547 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646557 2577 flags.go:64] FLAG: --register-node="true" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646560 2577 flags.go:64] FLAG: --register-with-taints="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646563 2577 flags.go:64] FLAG: --registry-burst="10" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646566 2577 flags.go:64] FLAG: --registry-qps="5" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646569 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646571 2577 flags.go:64] FLAG: --reserved-memory="" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646575 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646578 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646580 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646583 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 22:00:14.654292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646586 2577 flags.go:64] FLAG: --runonce="false" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646588 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646592 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646595 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646598 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646601 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646604 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646606 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646609 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646612 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646615 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646618 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646620 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646623 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646626 2577 flags.go:64] FLAG: --system-cgroups="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646629 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646634 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646637 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646640 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646644 2577 flags.go:64] FLAG: --tls-min-version="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646648 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646651 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646653 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646662 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646665 2577 flags.go:64] FLAG: --v="2" Sep 18 22:00:14.654930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646669 2577 flags.go:64] FLAG: --version="false" Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646672 2577 flags.go:64] FLAG: --vmodule="" Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646676 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.646679 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646803 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646807 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646810 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646813 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646816 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646820 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646823 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646826 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646829 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646831 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646834 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646837 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646839 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646842 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646845 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646847 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.655537 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646851 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646855 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646858 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646860 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646863 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646866 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646868 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646872 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646875 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646878 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646880 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646884 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646887 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646889 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646891 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646894 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646896 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646899 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646902 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.656041 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646904 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646907 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646911 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646914 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646916 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646919 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646921 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646924 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646927 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646929 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646932 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646934 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646937 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646939 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646942 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646945 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646947 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646950 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646952 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646955 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.656496 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646958 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646962 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646965 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646967 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646970 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646973 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646975 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646978 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646980 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646983 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646985 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646989 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646993 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.646996 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647000 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647003 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647006 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647008 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647011 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.657061 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647013 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647016 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647018 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647021 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647024 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647026 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647029 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647031 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647034 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647038 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647041 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647044 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647046 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647051 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647054 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647057 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647059 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647062 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647065 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647068 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.657517 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647070 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647073 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647075 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647078 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.647081 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.647976 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.655746 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.655763 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.656655 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.656663 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656730 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656735 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656738 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656741 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656743 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.658020 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656746 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656749 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656752 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656754 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656757 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656759 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656762 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656765 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656767 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656770 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656772 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656775 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656777 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656780 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656782 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656801 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656804 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656807 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656811 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656814 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.658392 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656818 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656821 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656824 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656827 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656836 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656839 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656842 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656844 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656847 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656849 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656852 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656855 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656857 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656860 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656862 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656865 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656867 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656870 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656872 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656874 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.658899 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656877 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656879 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656882 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656884 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656887 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656889 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656891 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656895 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656898 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656901 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656904 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656907 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656910 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656913 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656916 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656919 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656922 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656930 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656933 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.659399 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656936 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656939 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656942 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656944 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656947 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656949 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656952 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656955 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656957 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656959 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656962 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656964 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656967 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656969 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656971 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656974 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656976 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656979 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656981 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656984 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.659872 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656986 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656988 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656991 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656993 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.656997 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657000 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657003 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657005 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657008 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657011 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657014 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657022 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657025 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657028 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.660367 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657030 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.657035 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657191 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657196 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657199 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657202 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657205 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657209 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657212 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657215 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657217 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657220 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657223 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657226 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 22:00:14.660732 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657228 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657231 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657233 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657236 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657238 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657241 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657244 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657246 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657248 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657251 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657253 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657256 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657258 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657261 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657264 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657266 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657274 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657277 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657279 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657282 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 22:00:14.661149 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657286 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657289 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657292 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657294 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657297 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657299 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657302 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657304 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657307 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657309 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657312 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657314 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657317 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657319 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657321 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657324 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657326 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657329 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657331 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657334 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 22:00:14.661622 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657336 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657338 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657340 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657343 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657345 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657348 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657351 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657353 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657356 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657364 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657366 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657369 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657371 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657375 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657378 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657380 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657383 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657385 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657388 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 22:00:14.662147 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657390 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657393 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657395 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657397 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657400 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657402 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657405 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657407 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657409 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657412 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657414 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657416 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657419 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657421 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657424 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657426 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657429 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657431 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657434 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657436 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 22:00:14.662604 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657439 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657441 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657444 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657453 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657455 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657458 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657460 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.657463 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.657467 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.657608 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.659985 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.661110 2577 server.go:1019] "Starting client certificate rotation" Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.661217 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 22:00:14.663085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.662093 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 22:00:14.663865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.663852 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 22:00:14.665459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.665441 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 22:00:14.679826 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.679809 2577 log.go:25] "Validated CRI v1 runtime API" Sep 18 22:00:14.681589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.681573 2577 log.go:25] "Validated CRI v1 image API" Sep 18 22:00:14.683304 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.683289 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 22:00:14.687474 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.687456 2577 fs.go:135] Filesystem UUIDs: map[777ccff2-d9a1-4ea6-91ef-4efabc73c7a3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cd0b5e1a-e35a-4a80-b056-951dab7ed179:/dev/nvme0n1p3] Sep 18 22:00:14.687545 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.687474 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 22:00:14.692364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.692258 2577 manager.go:216] Machine: {Timestamp:2026-09-18 22:00:14.691028726 +0000 UTC m=+0.362526688 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099788 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db0b7dd57ef212b6d74606368414c SystemUUID:ec2db0b7-dd57-ef21-2b6d-74606368414c BootID:0564601b-e25e-4ee8-8001-44b1f8df66ac Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:29:ac:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:29:ac:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ef:ec:bc:20:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 22:00:14.692445 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.692366 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 22:00:14.692445 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.692436 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 22:00:14.693826 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.693781 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 22:00:14.693967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.693830 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 22:00:14.695555 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.695545 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 22:00:14.695592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.695557 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 22:00:14.695592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.695565 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 22:00:14.696465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.696455 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 22:00:14.698292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.698282 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 22:00:14.703782 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703770 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 18 22:00:14.703919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703909 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 22:00:14.703954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703938 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 18 22:00:14.703954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703948 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 22:00:14.704017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703961 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 22:00:14.704017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703970 2577 kubelet.go:398] "Adding apiserver pod source" Sep 18 22:00:14.704017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.703979 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 22:00:14.704343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.704328 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 22:00:14.704403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.704352 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 22:00:14.706859 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.706838 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 22:00:14.706953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.706910 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 22:00:14.709979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.709964 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 22:00:14.710042 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.709995 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 22:00:14.712938 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712925 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712945 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712951 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712956 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712963 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712969 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 22:00:14.712980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712982 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 22:00:14.713141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712988 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 22:00:14.713141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.712999 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 22:00:14.713141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.713004 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 22:00:14.713141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.713012 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 22:00:14.713141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.713017 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 22:00:14.714811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.714800 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 22:00:14.714811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.714811 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 22:00:14.716487 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.716465 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 22:00:14.717046 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.717025 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 22:00:14.718627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.718613 2577 server.go:1268] "Started kubelet" Sep 18 22:00:14.718774 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.718740 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 22:00:14.718935 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.718747 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 22:00:14.719046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.718958 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 22:00:14.719477 ip-10-0-133-172 systemd[1]: Started Kubernetes Kubelet. Sep 18 22:00:14.722387 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.722367 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 22:00:14.722899 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.722877 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 18 22:00:14.724854 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.724833 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 22:00:14.730501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.730484 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 22:00:14.731958 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.730761 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-172.ec2.internal.18d689411f5f90f2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-172.ec2.internal,UID:ip-10-0-133-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-172.ec2.internal,},FirstTimestamp:2026-09-18 22:00:14.718587122 +0000 UTC m=+0.390085083,LastTimestamp:2026-09-18 22:00:14.718587122 +0000 UTC m=+0.390085083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-172.ec2.internal,}" Sep 18 22:00:14.733847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.733826 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 22:00:14.734513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.734499 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 22:00:14.734815 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.734631 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:14.735270 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735252 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 22:00:14.735270 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735272 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735254 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735370 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735377 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735420 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735437 2577 factory.go:55] Registering systemd factory Sep 18 22:00:14.735465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735446 2577 factory.go:223] Registration of the systemd container factory successfully Sep 18 22:00:14.735735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735691 2577 factory.go:153] Registering CRI-O factory Sep 18 22:00:14.735735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735702 2577 factory.go:223] Registration of the crio container factory successfully Sep 18 22:00:14.735735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735717 2577 factory.go:103] Registering Raw factory Sep 18 22:00:14.735735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.735730 2577 manager.go:1196] Started watching for new ooms in manager Sep 18 22:00:14.736162 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.736148 2577 manager.go:349] Starting recovery of all containers Sep 18 22:00:14.738157 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.738106 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 22:00:14.738294 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.738269 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 22:00:14.741382 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:14.741362 2577 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max: no such device Sep 18 22:00:14.745233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.745214 2577 manager.go:354] Recovery completed Sep 18 22:00:14.748989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.748975 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.750738 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.750715 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.750815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.750751 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.750815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.750760 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.751305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751292 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 22:00:14.751364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751307 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 22:00:14.751364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751293 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.751364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751325 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 18 22:00:14.751949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751936 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.752007 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751959 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.752007 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.751969 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.753213 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.753123 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-172.ec2.internal.18d68941214a29b0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-172.ec2.internal,UID:ip-10-0-133-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-172.ec2.internal,},FirstTimestamp:2026-09-18 22:00:14.750738864 +0000 UTC m=+0.422236828,LastTimestamp:2026-09-18 22:00:14.750738864 +0000 UTC m=+0.422236828,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-172.ec2.internal,}" Sep 18 22:00:14.753471 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.753458 2577 policy_none.go:49] "None policy: Start" Sep 18 22:00:14.753528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.753477 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 22:00:14.753528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.753490 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 22:00:14.755406 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.755395 2577 policy_none.go:47] "Start" Sep 18 22:00:14.763952 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.763872 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-172.ec2.internal.18d68941214a6a8f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-172.ec2.internal,UID:ip-10-0-133-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-172.ec2.internal,},FirstTimestamp:2026-09-18 22:00:14.750755471 +0000 UTC m=+0.422253432,LastTimestamp:2026-09-18 22:00:14.750755471 +0000 UTC m=+0.422253432,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-172.ec2.internal,}" Sep 18 22:00:14.765304 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.765286 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqg66" Sep 18 22:00:14.772894 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.772837 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-172.ec2.internal.18d68941214a8cf6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-172.ec2.internal,UID:ip-10-0-133-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-172.ec2.internal,},FirstTimestamp:2026-09-18 22:00:14.750764278 +0000 UTC m=+0.422262239,LastTimestamp:2026-09-18 22:00:14.750764278 +0000 UTC m=+0.422262239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-172.ec2.internal,}" Sep 18 22:00:14.774081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.774062 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqg66" Sep 18 22:00:14.798323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798308 2577 manager.go:337] "Starting Device Plugin manager" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.798350 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798364 2577 server.go:87] "Starting device plugin registration server" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798615 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798625 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798734 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798852 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.798861 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.799258 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 22:00:14.815625 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.799309 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:14.840274 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.840243 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 22:00:14.841474 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.841459 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 22:00:14.841532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.841505 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 22:00:14.841532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.841522 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 22:00:14.841620 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.841553 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 22:00:14.844078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.844056 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:14.899324 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.899252 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.900118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.900104 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.900189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.900130 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.900189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.900141 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.900189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.900162 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.911647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.911631 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.911723 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.911656 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-172.ec2.internal\": node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:14.942488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.942463 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal"] Sep 18 22:00:14.942562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.942522 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.943257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.943242 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.943320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.943270 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.943320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.943283 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.945646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.945634 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.945778 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.945764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.945836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.945810 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.946307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946294 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.946367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946321 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.946367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946333 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.946964 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946948 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.947017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946982 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.947017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.946995 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.949225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.949211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.949285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.949234 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:14.949893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.949877 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:14.949953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.949907 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:14.949953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:14.949921 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:14.962097 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.962079 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:14.979332 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.979313 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-172.ec2.internal\" not found" node="ip-10-0-133-172.ec2.internal" Sep 18 22:00:14.983673 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:14.983658 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-172.ec2.internal\" not found" node="ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.037901 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.037880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.037993 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.037905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.037993 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.037921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ca949cbcee023acb5828904ff20c38f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-172.ec2.internal\" (UID: \"4ca949cbcee023acb5828904ff20c38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.062472 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.062454 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.138942 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.138916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.139024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.138952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.139024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.138968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ca949cbcee023acb5828904ff20c38f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-172.ec2.internal\" (UID: \"4ca949cbcee023acb5828904ff20c38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.139024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.139014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.139024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.139019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d95d21ef9d67890084e6ca3270a80d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal\" (UID: \"c9d95d21ef9d67890084e6ca3270a80d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.139161 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.139049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ca949cbcee023acb5828904ff20c38f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-172.ec2.internal\" (UID: \"4ca949cbcee023acb5828904ff20c38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.163064 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.163024 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.263708 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.263679 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.283848 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.283829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.287694 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.287680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:15.363908 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.363882 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.464436 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.464409 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.548719 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.548697 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:15.564918 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.564898 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.660483 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.660464 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 22:00:15.660881 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.660584 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 22:00:15.660881 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.660620 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 22:00:15.665689 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.665669 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.766801 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.766729 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.782889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.782861 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:55:15 +0000 UTC" deadline="2028-05-15 04:58:01.980622089 +0000 UTC" Sep 18 22:00:15.782889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.782883 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14502h57m46.197742061s" Sep 18 22:00:15.867134 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.867111 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:15.951590 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.951570 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 22:00:15.953131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.953111 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientMemory" Sep 18 22:00:15.953209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.953145 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 22:00:15.953209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:15.953156 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeHasSufficientPID" Sep 18 22:00:15.967554 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:15.967533 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:16.033465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.033409 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:16.068450 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.068426 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-172.ec2.internal\" not found" Sep 18 22:00:16.112114 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:16.112079 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca949cbcee023acb5828904ff20c38f.slice/crio-c7eb85acb44fdbf796dfc616fccffda0ac5cf53f1f1d11d7c7412a649f2fc4d6 WatchSource:0}: Error finding container c7eb85acb44fdbf796dfc616fccffda0ac5cf53f1f1d11d7c7412a649f2fc4d6: Status 404 returned error can't find the container with id c7eb85acb44fdbf796dfc616fccffda0ac5cf53f1f1d11d7c7412a649f2fc4d6 Sep 18 22:00:16.112718 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:16.112695 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d95d21ef9d67890084e6ca3270a80d.slice/crio-76715b59fd82ac3676dd2cf0b44e2356e59082537d3bc54af645234049bf76a0 WatchSource:0}: Error finding container 76715b59fd82ac3676dd2cf0b44e2356e59082537d3bc54af645234049bf76a0: Status 404 returned error can't find the container with id 76715b59fd82ac3676dd2cf0b44e2356e59082537d3bc54af645234049bf76a0 Sep 18 22:00:16.117970 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.117951 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:00:16.128577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.128555 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:16.135620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.135603 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" Sep 18 22:00:16.146021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.146006 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 22:00:16.146918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.146904 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" Sep 18 22:00:16.168203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.168185 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 22:00:16.705053 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.705030 2577 apiserver.go:52] "Watching apiserver" Sep 18 22:00:16.717610 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.717585 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 22:00:16.718069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.718046 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dmkfl","kube-system/konnectivity-agent-2n8hq","openshift-dns/node-resolver-s6wxg","openshift-multus/multus-fjswg","openshift-multus/network-metrics-daemon-v98jj","openshift-network-diagnostics/network-check-target-stszb","openshift-network-operator/iptables-alerter-g9c4k","kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl","openshift-cluster-node-tuning-operator/tuned-4c5g7","openshift-image-registry/node-ca-4qltq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal","openshift-multus/multus-additional-cni-plugins-l29z8","openshift-ovn-kubernetes/ovnkube-node-zgcxw"] Sep 18 22:00:16.720851 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.720815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.724433 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.723994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.727282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.727263 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.727725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.727704 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 22:00:16.727848 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.727759 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.727958 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.727937 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.728097 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.728080 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x5rmr\"" Sep 18 22:00:16.729563 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.729539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.731398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.731378 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.731599 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.731567 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4f49d\"" Sep 18 22:00:16.731765 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.731745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.732321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.732302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.732417 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.732379 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:16.733664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.733645 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 22:00:16.733947 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.733931 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.734255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.734236 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qrgll\"" Sep 18 22:00:16.734971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.734507 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 22:00:16.736337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.736062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:16.736337 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.736131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:16.736502 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.736361 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 22:00:16.737235 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.737018 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 22:00:16.737552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.737518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.738006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.737988 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 22:00:16.738428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.738406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.741695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.741667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.741786 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.741748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:16.742456 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.742430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4tr6w\"" Sep 18 22:00:16.742938 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.742919 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.743153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.743136 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 22:00:16.744096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.743347 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.744096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.743719 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vks6z\"" Sep 18 22:00:16.744096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.743743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.746890 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.746867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.746841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-multus\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-multus-daemon-config\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-multus-certs\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-cnibin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-hostroot\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtz8q\" (UniqueName: \"kubernetes.io/projected/520996bd-c31a-4981-a7e8-c23d73169878-kube-api-access-rtz8q\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5jj\" (UniqueName: \"kubernetes.io/projected/6b80a67d-582a-495f-9985-2b0bbbacde46-kube-api-access-rr5jj\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.747275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-binary-copy\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22lf\" (UniqueName: \"kubernetes.io/projected/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-kube-api-access-p22lf\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-socket-dir-parent\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-kubelet\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-etc-kubernetes\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-system-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-k8s-cni-cncf-io\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-bin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c0ce99-f972-4d13-b924-6e79457dd23f-iptables-alerter-script\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c0ce99-f972-4d13-b924-6e79457dd23f-host-slash\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cnibin\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-tmp-dir\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vnr\" (UniqueName: \"kubernetes.io/projected/51c0ce99-f972-4d13-b924-6e79457dd23f-kube-api-access-b4vnr\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.747776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-system-cni-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62df9\" (UniqueName: \"kubernetes.io/projected/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-kube-api-access-62df9\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-cni-binary-copy\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-conf-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-hosts-file\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.748390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.747901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-netns\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.750693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.749921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.750794 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.750745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.750898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.750767 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 22:00:16.750898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.750867 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.750898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.750856 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rmz6t\"" Sep 18 22:00:16.751267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.751238 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t2j2c\"" Sep 18 22:00:16.752553 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.752365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.759595 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.759571 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 22:00:16.759995 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.759801 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.760140 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.760201 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.760305 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.760394 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.760404 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m8tkh\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.761123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.761158 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.761249 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 22:00:16.761328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.761161 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 22:00:16.761913 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.761542 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cq8fn\"" Sep 18 22:00:16.765439 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.765416 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 22:00:16.807732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.807709 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmctd" Sep 18 22:00:16.831889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.831866 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmctd" Sep 18 22:00:16.837000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.836932 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 22:00:16.847505 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.847400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" event={"ID":"c9d95d21ef9d67890084e6ca3270a80d","Type":"ContainerStarted","Data":"76715b59fd82ac3676dd2cf0b44e2356e59082537d3bc54af645234049bf76a0"} Sep 18 22:00:16.848100 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-netns\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-multus\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-multus-daemon-config\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-systemd-units\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5jj\" (UniqueName: \"kubernetes.io/projected/6b80a67d-582a-495f-9985-2b0bbbacde46-kube-api-access-rr5jj\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-netd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/205cbc85-8170-4141-93f4-5ee0757a47bb-ovn-node-metrics-cert\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-binary-copy\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22lf\" (UniqueName: \"kubernetes.io/projected/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-kube-api-access-p22lf\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-socket-dir-parent\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-kubelet\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysconfig\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-kubernetes\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-conf\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-systemd\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-sys\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-lib-modules\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-ovn\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-config\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-system-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-socket-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-netns\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.848949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-multus\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c0ce99-f972-4d13-b924-6e79457dd23f-iptables-alerter-script\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c0ce99-f972-4d13-b924-6e79457dd23f-host-slash\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.848786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-multus-daemon-config\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cnibin\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-tmp-dir\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-kubelet-config\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-kubelet\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxpj\" (UniqueName: \"kubernetes.io/projected/205cbc85-8170-4141-93f4-5ee0757a47bb-kube-api-access-4sxpj\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-cni-binary-copy\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-sys-fs\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c0ce99-f972-4d13-b924-6e79457dd23f-iptables-alerter-script\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-dbus\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-kubelet\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-socket-dir-parent\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-run\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.849811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-host\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cnibin\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-slash\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-netns\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-systemd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-log-socket\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-bin\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.849974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-conf-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-hosts-file\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-multus-certs\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdcn\" (UniqueName: \"kubernetes.io/projected/00582bdc-45a3-4c1e-b623-04234b62cbd7-kube-api-access-wcdcn\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-env-overrides\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-cnibin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-hostroot\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtz8q\" (UniqueName: \"kubernetes.io/projected/520996bd-c31a-4981-a7e8-c23d73169878-kube-api-access-rtz8q\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-etc-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-etc-kubernetes\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.850651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-modprobe-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-var-lib-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-script-lib\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59c036de-03e7-45e5-bf69-0f0c829f65b4-konnectivity-ca\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf279\" (UniqueName: \"kubernetes.io/projected/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-kube-api-access-xf279\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-k8s-cni-cncf-io\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-bin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-tmp-dir\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6gd\" (UniqueName: \"kubernetes.io/projected/684d4849-13bd-42d3-b166-edbbc8e82092-kube-api-access-nt6gd\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-node-log\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-tmp\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-serviceca\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnr\" (UniqueName: \"kubernetes.io/projected/51c0ce99-f972-4d13-b924-6e79457dd23f-kube-api-access-b4vnr\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-system-cni-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.851396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-var-lib-cni-bin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62df9\" (UniqueName: \"kubernetes.io/projected/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-kube-api-access-62df9\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c0ce99-f972-4d13-b924-6e79457dd23f-host-slash\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59c036de-03e7-45e5-bf69-0f0c829f65b4-agent-certs\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.850990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-var-lib-kubelet\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-tuned\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-registration-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-device-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-host\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-multus-conf-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-hosts-file\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.851262 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-multus-certs\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.851324 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:17.351303671 +0000 UTC m=+3.022801643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-cnibin\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.852204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-host-run-k8s-cni-cncf-io\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" event={"ID":"4ca949cbcee023acb5828904ff20c38f","Type":"ContainerStarted","Data":"c7eb85acb44fdbf796dfc616fccffda0ac5cf53f1f1d11d7c7412a649f2fc4d6"} Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-system-cni-dir\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.851899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-system-cni-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.852085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/520996bd-c31a-4981-a7e8-c23d73169878-cni-binary-copy\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.852137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-hostroot\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.852282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-cni-binary-copy\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.852369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520996bd-c31a-4981-a7e8-c23d73169878-etc-kubernetes\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.853078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.852389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.853565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.853316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.870962 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.870936 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 22:00:16.871557 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.871526 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:16.871557 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.871556 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:16.871698 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.871572 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:16.871698 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.871630 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:17.37161774 +0000 UTC m=+3.043115690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:16.878035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.877969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5jj\" (UniqueName: \"kubernetes.io/projected/6b80a67d-582a-495f-9985-2b0bbbacde46-kube-api-access-rr5jj\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:16.878035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.878016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vnr\" (UniqueName: \"kubernetes.io/projected/51c0ce99-f972-4d13-b924-6e79457dd23f-kube-api-access-b4vnr\") pod \"iptables-alerter-g9c4k\" (UID: \"51c0ce99-f972-4d13-b924-6e79457dd23f\") " pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:16.882815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.882792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtz8q\" (UniqueName: \"kubernetes.io/projected/520996bd-c31a-4981-a7e8-c23d73169878-kube-api-access-rtz8q\") pod \"multus-fjswg\" (UID: \"520996bd-c31a-4981-a7e8-c23d73169878\") " pod="openshift-multus/multus-fjswg" Sep 18 22:00:16.886120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.886076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22lf\" (UniqueName: \"kubernetes.io/projected/96d1fbbd-29b7-4e76-b52e-b504c0ca9602-kube-api-access-p22lf\") pod \"multus-additional-cni-plugins-l29z8\" (UID: \"96d1fbbd-29b7-4e76-b52e-b504c0ca9602\") " pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:16.887961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.887916 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62df9\" (UniqueName: \"kubernetes.io/projected/33ac72fd-379b-4bc4-9fec-7942cf4ee26b-kube-api-access-62df9\") pod \"node-resolver-s6wxg\" (UID: \"33ac72fd-379b-4bc4-9fec-7942cf4ee26b\") " pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:16.951906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.951673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-netd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.951933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/205cbc85-8170-4141-93f4-5ee0757a47bb-ovn-node-metrics-cert\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.951964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysconfig\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.951990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-kubernetes\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-conf\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-systemd\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-sys\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-lib-modules\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-ovn\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-config\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-socket-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-kubelet-config\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-kubelet\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxpj\" (UniqueName: \"kubernetes.io/projected/205cbc85-8170-4141-93f4-5ee0757a47bb-kube-api-access-4sxpj\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-sys-fs\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-dbus\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-run\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-host\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-slash\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-netns\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-systemd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-log-socket\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-bin\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952611 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-systemd\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdcn\" (UniqueName: \"kubernetes.io/projected/00582bdc-45a3-4c1e-b623-04234b62cbd7-kube-api-access-wcdcn\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-env-overrides\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-sys\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-etc-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.952783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-modprobe-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-var-lib-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-lib-modules\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-script-lib\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59c036de-03e7-45e5-bf69-0f0c829f65b4-konnectivity-ca\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf279\" (UniqueName: \"kubernetes.io/projected/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-kube-api-access-xf279\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6gd\" (UniqueName: \"kubernetes.io/projected/684d4849-13bd-42d3-b166-edbbc8e82092-kube-api-access-nt6gd\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-node-log\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952941 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-socket-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.951860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-netd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.953069 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:16.953120 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:17.453103085 +0000 UTC m=+3.124601049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-kubelet-config\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-kubelet\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.953698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-tmp\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-serviceca\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59c036de-03e7-45e5-bf69-0f0c829f65b4-agent-certs\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-var-lib-kubelet\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-tuned\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-registration-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-device-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-host\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-systemd-units\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysconfig\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-kubernetes\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.953995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-sysctl-conf\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.954577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954103 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-registration-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954152 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-device-dir\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954191 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-host\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-systemd-units\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/684d4849-13bd-42d3-b166-edbbc8e82092-sys-fs\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d0405393-1b0b-446e-b958-231c5d2b0799-dbus\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-run\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-serviceca\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-host\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-slash\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-run-netns\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-systemd\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-log-socket\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.954714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-host-cni-bin\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-modprobe-d\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-var-lib-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.955466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-env-overrides\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-etc-openvswitch\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-script-lib\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.955757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00582bdc-45a3-4c1e-b623-04234b62cbd7-var-lib-kubelet\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.956349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.956255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/205cbc85-8170-4141-93f4-5ee0757a47bb-ovn-node-metrics-cert\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.956281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59c036de-03e7-45e5-bf69-0f0c829f65b4-konnectivity-ca\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.956625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.952807 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-run-ovn\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.956731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/205cbc85-8170-4141-93f4-5ee0757a47bb-ovnkube-config\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.956911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.956888 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/205cbc85-8170-4141-93f4-5ee0757a47bb-node-log\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.958844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.958818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-tmp\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.959475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.959459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59c036de-03e7-45e5-bf69-0f0c829f65b4-agent-certs\") pod \"konnectivity-agent-2n8hq\" (UID: \"59c036de-03e7-45e5-bf69-0f0c829f65b4\") " pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:16.959630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.959613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00582bdc-45a3-4c1e-b623-04234b62cbd7-etc-tuned\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.970387 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.970362 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxpj\" (UniqueName: \"kubernetes.io/projected/205cbc85-8170-4141-93f4-5ee0757a47bb-kube-api-access-4sxpj\") pod \"ovnkube-node-zgcxw\" (UID: \"205cbc85-8170-4141-93f4-5ee0757a47bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:16.975896 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.975850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf279\" (UniqueName: \"kubernetes.io/projected/c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92-kube-api-access-xf279\") pod \"node-ca-4qltq\" (UID: \"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92\") " pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:16.976681 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.976661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdcn\" (UniqueName: \"kubernetes.io/projected/00582bdc-45a3-4c1e-b623-04234b62cbd7-kube-api-access-wcdcn\") pod \"tuned-4c5g7\" (UID: \"00582bdc-45a3-4c1e-b623-04234b62cbd7\") " pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:16.977754 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:16.977733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6gd\" (UniqueName: \"kubernetes.io/projected/684d4849-13bd-42d3-b166-edbbc8e82092-kube-api-access-nt6gd\") pod \"aws-ebs-csi-driver-node-tbfcl\" (UID: \"684d4849-13bd-42d3-b166-edbbc8e82092\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:17.040921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.040884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g9c4k" Sep 18 22:00:17.050069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.050032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l29z8" Sep 18 22:00:17.054083 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.053758 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c0ce99_f972_4d13_b924_6e79457dd23f.slice/crio-0c37025ff5608f13f5b78f2e9d7594a8f911d33219d38cf079a1883b4ed755df WatchSource:0}: Error finding container 0c37025ff5608f13f5b78f2e9d7594a8f911d33219d38cf079a1883b4ed755df: Status 404 returned error can't find the container with id 0c37025ff5608f13f5b78f2e9d7594a8f911d33219d38cf079a1883b4ed755df Sep 18 22:00:17.060403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.060379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6wxg" Sep 18 22:00:17.063513 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.063390 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d1fbbd_29b7_4e76_b52e_b504c0ca9602.slice/crio-5792a81ab46fb0def5e2146e214d00a7be0fae88765a88658afc97fd9350816e WatchSource:0}: Error finding container 5792a81ab46fb0def5e2146e214d00a7be0fae88765a88658afc97fd9350816e: Status 404 returned error can't find the container with id 5792a81ab46fb0def5e2146e214d00a7be0fae88765a88658afc97fd9350816e Sep 18 22:00:17.068780 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.068755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjswg" Sep 18 22:00:17.070452 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.070426 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ac72fd_379b_4bc4_9fec_7942cf4ee26b.slice/crio-ac16d4f5e735efa3e03b0909cb2a093995f11308dd627e0d6d72bb5196dde204 WatchSource:0}: Error finding container ac16d4f5e735efa3e03b0909cb2a093995f11308dd627e0d6d72bb5196dde204: Status 404 returned error can't find the container with id ac16d4f5e735efa3e03b0909cb2a093995f11308dd627e0d6d72bb5196dde204 Sep 18 22:00:17.077305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.077275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" Sep 18 22:00:17.079510 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.079486 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520996bd_c31a_4981_a7e8_c23d73169878.slice/crio-341cbfcb3d278a5851e589c7d3b0d792b655277c2edb2fe0e5f532244b72547e WatchSource:0}: Error finding container 341cbfcb3d278a5851e589c7d3b0d792b655277c2edb2fe0e5f532244b72547e: Status 404 returned error can't find the container with id 341cbfcb3d278a5851e589c7d3b0d792b655277c2edb2fe0e5f532244b72547e Sep 18 22:00:17.086081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.086065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:17.089198 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.089173 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684d4849_13bd_42d3_b166_edbbc8e82092.slice/crio-9ac32dae1e1cae38b040a4e05c8199546983b1152c2455962b2be06d32ba28dd WatchSource:0}: Error finding container 9ac32dae1e1cae38b040a4e05c8199546983b1152c2455962b2be06d32ba28dd: Status 404 returned error can't find the container with id 9ac32dae1e1cae38b040a4e05c8199546983b1152c2455962b2be06d32ba28dd Sep 18 22:00:17.094593 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.094573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" Sep 18 22:00:17.097364 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.097338 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c036de_03e7_45e5_bf69_0f0c829f65b4.slice/crio-0518beea0a1c52ac616a66de16b573f73a2ccf689d19ef672766cd1c47323b98 WatchSource:0}: Error finding container 0518beea0a1c52ac616a66de16b573f73a2ccf689d19ef672766cd1c47323b98: Status 404 returned error can't find the container with id 0518beea0a1c52ac616a66de16b573f73a2ccf689d19ef672766cd1c47323b98 Sep 18 22:00:17.104147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.104121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qltq" Sep 18 22:00:17.106539 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.106514 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00582bdc_45a3_4c1e_b623_04234b62cbd7.slice/crio-7fc84786245ea24d1c0f151ab65c71866876f67dc0a688aca53d526cda800c25 WatchSource:0}: Error finding container 7fc84786245ea24d1c0f151ab65c71866876f67dc0a688aca53d526cda800c25: Status 404 returned error can't find the container with id 7fc84786245ea24d1c0f151ab65c71866876f67dc0a688aca53d526cda800c25 Sep 18 22:00:17.110391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.110372 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:17.119343 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.119318 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03b8ee4_9b2b_4e88_8d60_fa6d6e597f92.slice/crio-791e76875db736c8652b94fa53fef75d50518a0f336aa9e4dae75c6396ff7eb1 WatchSource:0}: Error finding container 791e76875db736c8652b94fa53fef75d50518a0f336aa9e4dae75c6396ff7eb1: Status 404 returned error can't find the container with id 791e76875db736c8652b94fa53fef75d50518a0f336aa9e4dae75c6396ff7eb1 Sep 18 22:00:17.121569 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:17.121548 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205cbc85_8170_4141_93f4_5ee0757a47bb.slice/crio-0c8b51a29f6585ac96f25bf33f7b4cacd54e88d451b9b36fa9402d5d02ccec3f WatchSource:0}: Error finding container 0c8b51a29f6585ac96f25bf33f7b4cacd54e88d451b9b36fa9402d5d02ccec3f: Status 404 returned error can't find the container with id 0c8b51a29f6585ac96f25bf33f7b4cacd54e88d451b9b36fa9402d5d02ccec3f Sep 18 22:00:17.262588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.262506 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:17.358501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.358460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:17.358683 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.358646 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:17.358728 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.358719 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:18.358701441 +0000 UTC m=+4.030199390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:17.459709 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.459668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:17.459924 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.459719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:17.459924 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.459904 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:17.459924 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.459927 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:17.460090 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.459937 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:17.460090 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.459951 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:17.460090 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.460007 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:18.459982493 +0000 UTC m=+4.131480470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:17.460090 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:17.460031 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:18.460019959 +0000 UTC m=+4.131517924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:17.835695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.835646 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:55:17 +0000 UTC" deadline="2028-02-14 20:10:25.864745601 +0000 UTC" Sep 18 22:00:17.835695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.835689 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12334h10m8.029061101s" Sep 18 22:00:17.854860 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.854825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerStarted","Data":"5792a81ab46fb0def5e2146e214d00a7be0fae88765a88658afc97fd9350816e"} Sep 18 22:00:17.856477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.856418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2n8hq" event={"ID":"59c036de-03e7-45e5-bf69-0f0c829f65b4","Type":"ContainerStarted","Data":"0518beea0a1c52ac616a66de16b573f73a2ccf689d19ef672766cd1c47323b98"} Sep 18 22:00:17.857717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.857688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" event={"ID":"684d4849-13bd-42d3-b166-edbbc8e82092","Type":"ContainerStarted","Data":"9ac32dae1e1cae38b040a4e05c8199546983b1152c2455962b2be06d32ba28dd"} Sep 18 22:00:17.859373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.859344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjswg" event={"ID":"520996bd-c31a-4981-a7e8-c23d73169878","Type":"ContainerStarted","Data":"341cbfcb3d278a5851e589c7d3b0d792b655277c2edb2fe0e5f532244b72547e"} Sep 18 22:00:17.861606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.861583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g9c4k" event={"ID":"51c0ce99-f972-4d13-b924-6e79457dd23f","Type":"ContainerStarted","Data":"0c37025ff5608f13f5b78f2e9d7594a8f911d33219d38cf079a1883b4ed755df"} Sep 18 22:00:17.863863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.863836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"0c8b51a29f6585ac96f25bf33f7b4cacd54e88d451b9b36fa9402d5d02ccec3f"} Sep 18 22:00:17.865202 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.865176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qltq" event={"ID":"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92","Type":"ContainerStarted","Data":"791e76875db736c8652b94fa53fef75d50518a0f336aa9e4dae75c6396ff7eb1"} Sep 18 22:00:17.866576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.866547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" event={"ID":"00582bdc-45a3-4c1e-b623-04234b62cbd7","Type":"ContainerStarted","Data":"7fc84786245ea24d1c0f151ab65c71866876f67dc0a688aca53d526cda800c25"} Sep 18 22:00:17.868603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.868577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6wxg" event={"ID":"33ac72fd-379b-4bc4-9fec-7942cf4ee26b","Type":"ContainerStarted","Data":"ac16d4f5e735efa3e03b0909cb2a093995f11308dd627e0d6d72bb5196dde204"} Sep 18 22:00:17.993124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:17.993091 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 22:00:18.365866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.365828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:18.366023 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.366001 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:18.366064 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.366056 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:20.366039422 +0000 UTC m=+6.037537371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:18.466818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.466773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:18.467020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.466894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:18.467085 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467039 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:18.467085 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467062 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:18.467085 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467075 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:18.467189 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467136 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:20.46711937 +0000 UTC m=+6.138617322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:18.467268 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467201 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:18.467268 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.467228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:20.467220239 +0000 UTC m=+6.138718189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:18.836267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.836024 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:55:17 +0000 UTC" deadline="2028-07-04 13:58:46.556817428 +0000 UTC" Sep 18 22:00:18.836267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.836063 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15711h58m27.720759603s" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.843007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.843142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.843562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.843659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.843811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:18.846380 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:18.843913 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:18.885781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.885705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" event={"ID":"4ca949cbcee023acb5828904ff20c38f","Type":"ContainerStarted","Data":"86322dc5bceeb4432c50c6913b9f72a624169e9f2d2e9c79dff54e941a2b52bc"} Sep 18 22:00:18.910630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:18.910571 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-172.ec2.internal" podStartSLOduration=2.910551117 podStartE2EDuration="2.910551117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:18.910045211 +0000 UTC m=+4.581543186" watchObservedRunningTime="2026-09-18 22:00:18.910551117 +0000 UTC m=+4.582049093" Sep 18 22:00:19.909442 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:19.909394 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" event={"ID":"c9d95d21ef9d67890084e6ca3270a80d","Type":"ContainerStarted","Data":"909cd751050e8f7eb5e6180b22c4b6f3cb683082b1bc9a210f6886903b787dcc"} Sep 18 22:00:20.388170 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.387538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:20.388170 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.387727 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:20.388170 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.387797 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:24.38777725 +0000 UTC m=+10.059275205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:20.488469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.488396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:20.488469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.488451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:20.488730 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.488578 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:20.488730 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.488642 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:24.488622618 +0000 UTC m=+10.160120569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:20.489273 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.489104 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:20.489273 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.489126 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:20.489273 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.489138 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:20.489273 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.489189 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:24.489173561 +0000 UTC m=+10.160671526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:20.844739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.844031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:20.844739 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.844163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:20.844739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.844572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:20.844739 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.844676 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:20.845130 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:20.844885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:20.845130 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:20.844987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:21.920266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:21.920174 2577 generic.go:356] "Generic (PLEG): container finished" podID="c9d95d21ef9d67890084e6ca3270a80d" containerID="909cd751050e8f7eb5e6180b22c4b6f3cb683082b1bc9a210f6886903b787dcc" exitCode=0 Sep 18 22:00:21.920266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:21.920225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" event={"ID":"c9d95d21ef9d67890084e6ca3270a80d","Type":"ContainerDied","Data":"909cd751050e8f7eb5e6180b22c4b6f3cb683082b1bc9a210f6886903b787dcc"} Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:22.844643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:22.844770 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:22.845146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:22.845248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:22.845291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:22.845410 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:22.845353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:24.424282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.424242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:24.424706 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.424410 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:24.424706 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.424466 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:32.424445516 +0000 UTC m=+18.095943481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.525005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.525047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525170 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525222 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:32.525205797 +0000 UTC m=+18.196703758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525585 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525607 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525619 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:24.525691 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.525661 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:32.52564805 +0000 UTC m=+18.197146001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:24.842834 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.842782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:24.842992 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.842905 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:24.843329 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.843311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:24.843445 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.843420 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:24.843581 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:24.843566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:24.843678 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:24.843649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:26.842513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:26.842474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:26.843067 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:26.842608 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:26.843067 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:26.843011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:26.843188 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:26.843113 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:26.843606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:26.843226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:26.843606 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:26.843296 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:28.844922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:28.844889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:28.845355 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:28.844897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:28.845355 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:28.845008 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:28.845355 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:28.845020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:28.845355 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:28.845085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:28.845355 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:28.845156 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:30.842175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:30.842140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:30.842175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:30.842169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:30.842648 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:30.842250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:30.842648 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:30.842262 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:30.842648 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:30.842340 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:30.842648 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:30.842424 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:32.484985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.484941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:32.485489 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.485128 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:32.485489 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.485209 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:48.485186544 +0000 UTC m=+34.156684513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:32.585955 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.585921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:32.586102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.585963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:32.586102 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586089 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:32.586209 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586114 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:32.586209 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586140 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:32.586209 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586155 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:32.586209 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586143 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:48.586125783 +0000 UTC m=+34.257623748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:32.586376 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.586217 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:00:48.586200381 +0000 UTC m=+34.257698342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.842050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.842064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:32.842046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.842187 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.842266 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:32.843822 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:32.842376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:34.843508 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:34.843281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:34.843970 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:34.843357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:34.843970 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:34.843595 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:34.843970 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:34.843710 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:34.843970 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:34.843388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:34.843970 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:34.843825 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:36.842660 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.842450 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:36.843223 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.842450 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:36.843223 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:36.842702 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:36.843223 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:36.842760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:36.843223 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.842522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:36.843223 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:36.842874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:36.951062 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.951036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" event={"ID":"c9d95d21ef9d67890084e6ca3270a80d","Type":"ContainerStarted","Data":"1dec5ad8de4181465cbf35628c27bec7d5e74a7356ecf6c74949e7327e7439ac"} Sep 18 22:00:36.953948 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.953923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"a717b59980d926d0a18a48f20b57e2da733211514f36f294d333e7997cde14f1"} Sep 18 22:00:36.954077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.953985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"efb3b7e2919730efd04a172cf352a460c76efce6cf4b802d27393c2eff2cf6c9"} Sep 18 22:00:36.954077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.954020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"dc07e372a8e0e56583e298c2e3060631a88a25f2d6b845a93ff4215654ed2e78"} Sep 18 22:00:36.954077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.954061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"3f3496ead1cb6dbc1aa831f6172715c25eb3c6668c2d9ad9d01281dde3a72de7"} Sep 18 22:00:36.954213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.954090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"937bcdd8628f417aea701fdd4f3b08b9612e4ec9e5a9d0682c428ee5e0ccf04c"} Sep 18 22:00:36.954213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.954117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"8a11650856cd9f180fa1a4f198e3637977eab5baee7e13deff0a8c41c80fc7c9"} Sep 18 22:00:36.955421 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.955393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qltq" event={"ID":"c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92","Type":"ContainerStarted","Data":"55a0f018a29549b9b8b27f4bd82d505a72e4edd64d20211815153fe15ae66c8d"} Sep 18 22:00:36.956872 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.956842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" event={"ID":"00582bdc-45a3-4c1e-b623-04234b62cbd7","Type":"ContainerStarted","Data":"3fca65e9c133cc654028d09c6e9a1576e89a581398f522cdd9c76e703026ef03"} Sep 18 22:00:36.958175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.958156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6wxg" event={"ID":"33ac72fd-379b-4bc4-9fec-7942cf4ee26b","Type":"ContainerStarted","Data":"224af753cce6257a26f02b89bbdecd9c332f608b7993aa8495ad25500bf8e05e"} Sep 18 22:00:36.959660 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.959640 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="49a320dca8551523277d6a62f77c9b4b388f6b79af70e31d0d449f2316c9a233" exitCode=0 Sep 18 22:00:36.959752 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.959698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"49a320dca8551523277d6a62f77c9b4b388f6b79af70e31d0d449f2316c9a233"} Sep 18 22:00:36.961287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.961269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2n8hq" event={"ID":"59c036de-03e7-45e5-bf69-0f0c829f65b4","Type":"ContainerStarted","Data":"a59aa01655f9808e1cdf0d1af7a0696214ea97c0da05f7ec8281cbbf0316efec"} Sep 18 22:00:36.962757 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.962740 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" event={"ID":"684d4849-13bd-42d3-b166-edbbc8e82092","Type":"ContainerStarted","Data":"31d302acacc48a189e531c23cdece407acf98e38cf746ebfd4d2dde61aaf0941"} Sep 18 22:00:36.964191 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.964175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjswg" event={"ID":"520996bd-c31a-4981-a7e8-c23d73169878","Type":"ContainerStarted","Data":"af4398c4583b39cd85d53d96d2cec36374522326c48fd63b430f663ae0890aae"} Sep 18 22:00:36.972513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:36.972481 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-172.ec2.internal" podStartSLOduration=20.972470962 podStartE2EDuration="20.972470962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:36.972020903 +0000 UTC m=+22.643518870" watchObservedRunningTime="2026-09-18 22:00:36.972470962 +0000 UTC m=+22.643968931" Sep 18 22:00:37.011147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.011117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:37.011565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.011543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:37.016871 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.016837 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4c5g7" podStartSLOduration=3.128653149 podStartE2EDuration="22.016826774s" totalImagesPullingTime="18.888173625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:36.993765827 +0000 UTC m=+22.665263794" watchObservedRunningTime="2026-09-18 22:00:37.016826774 +0000 UTC m=+22.688324745" Sep 18 22:00:37.051871 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.051838 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4qltq" podStartSLOduration=3.580644352 podStartE2EDuration="22.051827172s" totalImagesPullingTime="18.47118282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:37.017561998 +0000 UTC m=+22.689059969" watchObservedRunningTime="2026-09-18 22:00:37.051827172 +0000 UTC m=+22.723325141" Sep 18 22:00:37.052160 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.052140 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s6wxg" podStartSLOduration=3.5322432580000003 podStartE2EDuration="22.052134317s" totalImagesPullingTime="18.519891059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:37.051952473 +0000 UTC m=+22.723450425" watchObservedRunningTime="2026-09-18 22:00:37.052134317 +0000 UTC m=+22.723632284" Sep 18 22:00:37.113142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.113107 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2n8hq" podStartSLOduration=3.215925789 podStartE2EDuration="22.113095392s" totalImagesPullingTime="18.897169603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:37.076257661 +0000 UTC m=+22.747755641" watchObservedRunningTime="2026-09-18 22:00:37.113095392 +0000 UTC m=+22.784593359" Sep 18 22:00:37.157154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.157087 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fjswg" podStartSLOduration=2.994029474 podStartE2EDuration="22.157071814s" totalImagesPullingTime="19.16304234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:37.136079908 +0000 UTC m=+22.807577877" watchObservedRunningTime="2026-09-18 22:00:37.157071814 +0000 UTC m=+22.828569787" Sep 18 22:00:37.652600 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.652579 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 22:00:37.813429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.813286 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T22:00:37.652596933Z","UUID":"4b30ea3d-f0ff-4768-9d5b-7c545f5c15f7","Handler":null,"Name":"","Endpoint":""} Sep 18 22:00:37.814569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.814555 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 22:00:37.814649 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.814581 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 22:00:37.967765 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.967737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" event={"ID":"684d4849-13bd-42d3-b166-edbbc8e82092","Type":"ContainerStarted","Data":"88763991f861b38b36d429fffc4614e8560b3271bce684d8aeea43dabb83d1bc"} Sep 18 22:00:37.969275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.969254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g9c4k" event={"ID":"51c0ce99-f972-4d13-b924-6e79457dd23f","Type":"ContainerStarted","Data":"c781369446209fcb93b67db3ba6a37755b7684900e084bdfb61d473ed7b178d3"} Sep 18 22:00:37.970119 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.970087 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:37.970243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:37.970229 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2n8hq" Sep 18 22:00:38.003513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:38.003470 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g9c4k" podStartSLOduration=4.469497402 podStartE2EDuration="23.003458313s" totalImagesPullingTime="18.533960911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:38.00314217 +0000 UTC m=+23.674640141" watchObservedRunningTime="2026-09-18 22:00:38.003458313 +0000 UTC m=+23.674956282" Sep 18 22:00:38.842126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:38.842082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:38.842224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:38.842094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:38.842224 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:38.842189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:38.842337 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:38.842311 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:38.842379 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:38.842358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:38.842477 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:38.842456 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:38.974547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:38.974517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"dc2d50e87550abfe3b68b373f3477e1538945ae345403bbdf8576b57ff8e2d43"} Sep 18 22:00:39.979823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:39.979607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" event={"ID":"684d4849-13bd-42d3-b166-edbbc8e82092","Type":"ContainerStarted","Data":"71d80b67a15c9f80bc17d847d12e995700ca709e2ba4725ec6c43dd2f662b002"} Sep 18 22:00:40.002516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:40.002467 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbfcl" podStartSLOduration=3.599836219 podStartE2EDuration="25.002453122s" totalImagesPullingTime="21.402616903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:40.001431838 +0000 UTC m=+25.672929811" watchObservedRunningTime="2026-09-18 22:00:40.002453122 +0000 UTC m=+25.673951091" Sep 18 22:00:40.841929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:40.841896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:40.842115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:40.841896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:40.842115 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:40.842027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:40.842115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:40.841896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:40.842289 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:40.842113 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:40.842289 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:40.842195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:42.842724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.842568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:42.843132 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.842591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:42.843132 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:42.842826 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:42.843132 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:42.842909 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:42.843132 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.842591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:42.843132 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:42.843027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:42.990404 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.990369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" event={"ID":"205cbc85-8170-4141-93f4-5ee0757a47bb","Type":"ContainerStarted","Data":"1eafbc35fd2d32291734f3425c9c0e07ebc6ef2e72e78369c1b11ca14f3bc4e2"} Sep 18 22:00:42.990655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.990631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:42.990772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.990694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:42.992624 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:42.992601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerStarted","Data":"9547c07d68236b72b97fa152fb02851ef9ba5f0f5e8d2939c126f388fbcbe6e7"} Sep 18 22:00:43.005965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:43.005947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:43.021082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:43.021045 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" podStartSLOduration=9.10247066 podStartE2EDuration="28.021030689s" totalImagesPullingTime="18.918560029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:43.020919109 +0000 UTC m=+28.692417076" watchObservedRunningTime="2026-09-18 22:00:43.021030689 +0000 UTC m=+28.692528658" Sep 18 22:00:43.996881 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:43.996836 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="9547c07d68236b72b97fa152fb02851ef9ba5f0f5e8d2939c126f388fbcbe6e7" exitCode=0 Sep 18 22:00:43.997418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:43.996922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"9547c07d68236b72b97fa152fb02851ef9ba5f0f5e8d2939c126f388fbcbe6e7"} Sep 18 22:00:43.997418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:43.997344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:44.012477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.012462 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:00:44.456850 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.456823 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stszb"] Sep 18 22:00:44.457016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.456949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:44.457061 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:44.457028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:44.462339 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.462315 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v98jj"] Sep 18 22:00:44.462435 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.462417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:44.462506 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:44.462491 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:44.465136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.465104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dmkfl"] Sep 18 22:00:44.465223 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:44.465209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:44.465330 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:44.465313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:45.842066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:45.841921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:45.842543 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:45.841922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:45.842543 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:45.842137 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:45.842543 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:45.842226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:46.003956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:46.003924 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="7afbf9f75f9f29be4b72cf594a6518448a2c33651b8276082112b20fb2793117" exitCode=0 Sep 18 22:00:46.004083 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:46.004013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"7afbf9f75f9f29be4b72cf594a6518448a2c33651b8276082112b20fb2793117"} Sep 18 22:00:46.842217 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:46.842182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:46.842631 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:46.842315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:47.008671 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:47.008645 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="854e71c0c6c32963d35a174596c70d3455e397c6ce9401a6cb9ba4df26735677" exitCode=0 Sep 18 22:00:47.008771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:47.008703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"854e71c0c6c32963d35a174596c70d3455e397c6ce9401a6cb9ba4df26735677"} Sep 18 22:00:47.841835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:47.841801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:47.841835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:47.841831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:47.842071 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:47.841926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:47.842071 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:47.842038 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:48.501920 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:48.501881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:48.502359 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.502119 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:48.502359 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.502203 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.502181664 +0000 UTC m=+66.173679612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 22:00:48.602806 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:48.602752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:48.602971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:48.602822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:48.602971 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.602922 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:48.602971 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.602939 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 22:00:48.602971 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.602961 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 22:00:48.602971 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.602974 2577 projected.go:196] Error preparing data for projected volume kube-api-access-cb472 for pod openshift-network-diagnostics/network-check-target-stszb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:48.603236 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.602991 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret podName:d0405393-1b0b-446e-b958-231c5d2b0799 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.602971431 +0000 UTC m=+66.274469396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret") pod "global-pull-secret-syncer-dmkfl" (UID: "d0405393-1b0b-446e-b958-231c5d2b0799") : object "kube-system"/"original-pull-secret" not registered Sep 18 22:00:48.603236 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.603023 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472 podName:8dd12078-56f8-4276-8e34-cce23a44e7e4 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.603007188 +0000 UTC m=+66.274505142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb472" (UniqueName: "kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472") pod "network-check-target-stszb" (UID: "8dd12078-56f8-4276-8e34-cce23a44e7e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 22:00:48.846135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:48.846065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:48.846282 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:48.846172 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v98jj" podUID="6b80a67d-582a-495f-9985-2b0bbbacde46" Sep 18 22:00:49.842069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:49.842043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:49.842499 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:49.842150 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stszb" podUID="8dd12078-56f8-4276-8e34-cce23a44e7e4" Sep 18 22:00:49.842499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:49.842219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:49.842499 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:49.842341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dmkfl" podUID="d0405393-1b0b-446e-b958-231c5d2b0799" Sep 18 22:00:50.096567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.096491 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-172.ec2.internal" event="NodeReady" Sep 18 22:00:50.096727 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.096628 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 22:00:50.135344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.135319 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:00:50.142178 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.142151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.145173 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.145140 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 22:00:50.146316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.145935 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 22:00:50.146316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.146158 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m9bdj\"" Sep 18 22:00:50.146452 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.146431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 22:00:50.151906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.151888 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 22:00:50.152172 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.152145 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:00:50.154756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.154734 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bdkwr"] Sep 18 22:00:50.158218 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.158200 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d6zt4"] Sep 18 22:00:50.158343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.158327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.160854 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.160833 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 22:00:50.161120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.161103 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 22:00:50.161357 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.161338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdhvj\"" Sep 18 22:00:50.161670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.161655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.164695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.164673 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 22:00:50.164807 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.164766 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 22:00:50.164876 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.164855 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vdm9t\"" Sep 18 22:00:50.165580 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.165561 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 22:00:50.171102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.171081 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdkwr"] Sep 18 22:00:50.172017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.171994 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6zt4"] Sep 18 22:00:50.314528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34307b55-2153-4ea8-914a-9c5fbffe1deb-config-volume\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.314702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.314702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbqv\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqc5\" (UniqueName: \"kubernetes.io/projected/6e137694-e4b9-4aed-8de3-32e7f4984392-kube-api-access-9tqc5\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34307b55-2153-4ea8-914a-9c5fbffe1deb-tmp-dir\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk89v\" (UniqueName: \"kubernetes.io/projected/34307b55-2153-4ea8-914a-9c5fbffe1deb-kube-api-access-fk89v\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.314994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.314986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e137694-e4b9-4aed-8de3-32e7f4984392-cert\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.415831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.415744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.415831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.415778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34307b55-2153-4ea8-914a-9c5fbffe1deb-tmp-dir\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.416046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.415844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk89v\" (UniqueName: \"kubernetes.io/projected/34307b55-2153-4ea8-914a-9c5fbffe1deb-kube-api-access-fk89v\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.416046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.415860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.416046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.415891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.416203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416141 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.416203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34307b55-2153-4ea8-914a-9c5fbffe1deb-tmp-dir\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e137694-e4b9-4aed-8de3-32e7f4984392-cert\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34307b55-2153-4ea8-914a-9c5fbffe1deb-config-volume\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.416535 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.416595 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:00:50.916578067 +0000 UTC m=+36.588076015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.416637 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.416651 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbqv\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.416699 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:00:50.916686182 +0000 UTC m=+36.588184147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqc5\" (UniqueName: \"kubernetes.io/projected/6e137694-e4b9-4aed-8de3-32e7f4984392-kube-api-access-9tqc5\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.417017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.416959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34307b55-2153-4ea8-914a-9c5fbffe1deb-config-volume\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.417922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.417338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.421302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.421118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.421302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.421225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.421302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.421261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e137694-e4b9-4aed-8de3-32e7f4984392-cert\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.429832 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.429813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqc5\" (UniqueName: \"kubernetes.io/projected/6e137694-e4b9-4aed-8de3-32e7f4984392-kube-api-access-9tqc5\") pod \"ingress-canary-d6zt4\" (UID: \"6e137694-e4b9-4aed-8de3-32e7f4984392\") " pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.429987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.429970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.430084 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.430068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk89v\" (UniqueName: \"kubernetes.io/projected/34307b55-2153-4ea8-914a-9c5fbffe1deb-kube-api-access-fk89v\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.430122 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.430096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbqv\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.478321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.478296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6zt4" Sep 18 22:00:50.846125 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.846093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:00:50.849478 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.849453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 22:00:50.849593 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.849455 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27j86\"" Sep 18 22:00:50.920056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.920027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:50.920193 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:50.920060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:50.920309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.920190 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:00:50.920309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.920245 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:00:51.920230565 +0000 UTC m=+37.591728538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:00:50.920309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.920193 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:00:50.920309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.920289 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:00:50.920518 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:50.920349 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:00:51.920332855 +0000 UTC m=+37.591830803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:00:51.842015 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.841865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:00:51.842115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.841865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:00:51.845378 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.845356 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 22:00:51.845568 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.845548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 22:00:51.846172 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.846153 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 22:00:51.846495 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.846202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8l8bl\"" Sep 18 22:00:51.927045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.927014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:51.927045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:51.927048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:51.927266 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:51.927137 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:00:51.927266 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:51.927144 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:00:51.927266 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:51.927158 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:00:51.927266 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:51.927200 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:00:53.927184367 +0000 UTC m=+39.598682319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:00:51.927266 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:51.927217 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:00:53.927210455 +0000 UTC m=+39.598708411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:00:52.042637 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:52.042610 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6zt4"] Sep 18 22:00:52.046188 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:00:52.046161 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e137694_e4b9_4aed_8de3_32e7f4984392.slice/crio-1d0d9c8d2592df8a057a076e236d18426db8a5bc503bff7a268c83130843389f WatchSource:0}: Error finding container 1d0d9c8d2592df8a057a076e236d18426db8a5bc503bff7a268c83130843389f: Status 404 returned error can't find the container with id 1d0d9c8d2592df8a057a076e236d18426db8a5bc503bff7a268c83130843389f Sep 18 22:00:53.025873 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:53.025843 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="1a6ac568c0be7763efcd0a10da72a1ea9509539b862e57ae0ddfa845fa70d03d" exitCode=0 Sep 18 22:00:53.026467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:53.025921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"1a6ac568c0be7763efcd0a10da72a1ea9509539b862e57ae0ddfa845fa70d03d"} Sep 18 22:00:53.027535 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:53.027511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6zt4" event={"ID":"6e137694-e4b9-4aed-8de3-32e7f4984392","Type":"ContainerStarted","Data":"1d0d9c8d2592df8a057a076e236d18426db8a5bc503bff7a268c83130843389f"} Sep 18 22:00:53.943602 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:53.943548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:53.943602 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:53.943600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:53.943953 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:53.943706 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:00:53.943953 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:53.943706 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:00:53.943953 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:53.943770 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:00:57.943755845 +0000 UTC m=+43.615253796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:00:53.943953 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:53.943717 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:00:53.943953 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:53.943840 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:00:57.943828827 +0000 UTC m=+43.615326792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:00:54.033474 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:54.033447 2577 generic.go:356] "Generic (PLEG): container finished" podID="96d1fbbd-29b7-4e76-b52e-b504c0ca9602" containerID="a9895316044f52005d33968e45c5e3bba2cbb00e59f5aa03846082008dfcdebc" exitCode=0 Sep 18 22:00:54.033941 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:54.033489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerDied","Data":"a9895316044f52005d33968e45c5e3bba2cbb00e59f5aa03846082008dfcdebc"} Sep 18 22:00:54.458992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:54.458964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:54.593712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:54.593689 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:55.038517 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.038465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l29z8" event={"ID":"96d1fbbd-29b7-4e76-b52e-b504c0ca9602","Type":"ContainerStarted","Data":"36979eed06dcde5904427e3f8c9531afad1785d0fab68acf97ac8909e347d836"} Sep 18 22:00:55.039878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.039853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6zt4" event={"ID":"6e137694-e4b9-4aed-8de3-32e7f4984392","Type":"ContainerStarted","Data":"fc95f68d1766e2a7d360b350e2ac99c0e3312650bfe6088a1c7eea9e724de95d"} Sep 18 22:00:55.091255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.091206 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l29z8" podStartSLOduration=7.837950852 podStartE2EDuration="40.091189618s" totalImagesPullingTime="32.253238766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:55.08935545 +0000 UTC m=+40.760853419" watchObservedRunningTime="2026-09-18 22:00:55.091189618 +0000 UTC m=+40.762687590" Sep 18 22:00:55.120387 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.120347 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d6zt4" podStartSLOduration=2.614774566 podStartE2EDuration="5.120334745s" totalImagesPullingTime="2.505560179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:55.119574824 +0000 UTC m=+40.791072794" watchObservedRunningTime="2026-09-18 22:00:55.120334745 +0000 UTC m=+40.791832714" Sep 18 22:00:55.238925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.238882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:55.307585 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:55.307504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:57.024965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:57.024936 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:57.145387 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:57.145362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:00:57.972283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:57.972254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:00:57.972283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:00:57.972282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:00:57.972508 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:57.972396 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:00:57.972508 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:57.972406 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:00:57.972508 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:57.972409 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:00:57.972508 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:57.972456 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:01:05.97244328 +0000 UTC m=+51.643941228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:00:57.972508 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:00:57.972469 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:01:05.97246312 +0000 UTC m=+51.643961068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:01:01.415307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:01.415218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:01.517013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:01.516984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:01.594449 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:01.594423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:01.670776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:01.670715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:02.761072 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:02.761044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:02.861810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:02.861764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:03.697963 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:03.697933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:03.784956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:03.784929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:06.030487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:06.030458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:01:06.030487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:06.030489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:06.030923 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:06.030587 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:01:06.030923 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:06.030590 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:01:06.030923 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:06.030598 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:01:06.030923 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:06.030651 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:01:22.030635751 +0000 UTC m=+67.702133703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:01:06.030923 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:06.030665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:01:22.030658973 +0000 UTC m=+67.702156921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:01:13.172389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.172354 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8"] Sep 18 22:01:13.204888 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.204860 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph"] Sep 18 22:01:13.205012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.204961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.208747 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.208728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b4tfr\"" Sep 18 22:01:13.209193 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.209172 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 18 22:01:13.209289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.209205 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 18 22:01:13.213703 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.213690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 18 22:01:13.217462 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.217446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8"] Sep 18 22:01:13.217532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.217523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.233366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.233344 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph"] Sep 18 22:01:13.253803 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.253767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 22:01:13.254954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.254936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 22:01:13.255033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.254936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 22:01:13.255033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.254977 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 22:01:13.255033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.255010 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6jvbp\"" Sep 18 22:01:13.275465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.275442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq9p\" (UniqueName: \"kubernetes.io/projected/96343875-434b-4192-8130-794208b79619-kube-api-access-xpq9p\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.275539 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.275526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.275584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.275546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxv4\" (UniqueName: \"kubernetes.io/projected/9af01b02-674e-45bf-a307-d6d96a6ead9b-kube-api-access-dbxv4\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.275584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.275561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96343875-434b-4192-8130-794208b79619-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.275584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.275580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.376280 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96343875-434b-4192-8130-794208b79619-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.376374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.376374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq9p\" (UniqueName: \"kubernetes.io/projected/96343875-434b-4192-8130-794208b79619-kube-api-access-xpq9p\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.376487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.376487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxv4\" (UniqueName: \"kubernetes.io/projected/9af01b02-674e-45bf-a307-d6d96a6ead9b-kube-api-access-dbxv4\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.376487 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.376420 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:13.376618 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.376508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:13.876487784 +0000 UTC m=+59.547985750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:13.376618 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.376509 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 22:01:13.376618 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.376567 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls podName:9af01b02-674e-45bf-a307-d6d96a6ead9b nodeName:}" failed. No retries permitted until 2026-09-18 22:01:13.876554206 +0000 UTC m=+59.548052158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-nnfn8" (UID: "9af01b02-674e-45bf-a307-d6d96a6ead9b") : secret "samples-operator-tls" not found Sep 18 22:01:13.376982 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.376965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96343875-434b-4192-8130-794208b79619-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.385572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.385548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq9p\" (UniqueName: \"kubernetes.io/projected/96343875-434b-4192-8130-794208b79619-kube-api-access-xpq9p\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.392944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.392921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxv4\" (UniqueName: \"kubernetes.io/projected/9af01b02-674e-45bf-a307-d6d96a6ead9b-kube-api-access-dbxv4\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.440301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.440275 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:13.540430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.540408 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:13.879857 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.879772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:13.879857 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:13.879827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:13.880041 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.879910 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 22:01:13.880041 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.879920 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:13.880041 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.879970 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls podName:9af01b02-674e-45bf-a307-d6d96a6ead9b nodeName:}" failed. No retries permitted until 2026-09-18 22:01:14.879955919 +0000 UTC m=+60.551453883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-nnfn8" (UID: "9af01b02-674e-45bf-a307-d6d96a6ead9b") : secret "samples-operator-tls" not found Sep 18 22:01:13.880041 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:13.879984 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:14.879978168 +0000 UTC m=+60.551476116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:14.884422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:14.884384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:14.884422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:14.884421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:14.884780 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:14.884515 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 22:01:14.884780 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:14.884549 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:14.884780 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:14.884579 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls podName:9af01b02-674e-45bf-a307-d6d96a6ead9b nodeName:}" failed. No retries permitted until 2026-09-18 22:01:16.88456254 +0000 UTC m=+62.556060500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-nnfn8" (UID: "9af01b02-674e-45bf-a307-d6d96a6ead9b") : secret "samples-operator-tls" not found Sep 18 22:01:14.884780 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:14.884593 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:16.884587047 +0000 UTC m=+62.556084994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:15.726689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:15.726657 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:15.784541 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:15.784514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:16.014512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:16.014440 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zgcxw" Sep 18 22:01:16.180519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:16.180489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:16.243921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:16.243894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:16.898648 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:16.898611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:16.898648 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:16.898654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:16.898937 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:16.898763 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:16.898937 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:16.898765 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 22:01:16.898937 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:16.898857 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls podName:9af01b02-674e-45bf-a307-d6d96a6ead9b nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.898836347 +0000 UTC m=+66.570334311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-nnfn8" (UID: "9af01b02-674e-45bf-a307-d6d96a6ead9b") : secret "samples-operator-tls" not found Sep 18 22:01:16.898937 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:16.898875 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.898866814 +0000 UTC m=+66.570364770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:17.389570 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:17.389537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:17.525466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:17.525438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:17.890926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:17.890893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:18.000250 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:17.998070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:18.098360 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:18.098330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:19.365576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.365543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:19.487284 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.487259 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2xnq2"] Sep 18 22:01:19.487743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.487719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:19.491727 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.491710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.494407 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.494383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 22:01:19.494618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.494601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 22:01:19.494690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.494622 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 22:01:19.494690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.494682 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 22:01:19.495528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.495515 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wk4th\"" Sep 18 22:01:19.507603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.507582 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2xnq2"] Sep 18 22:01:19.517174 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.517154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/092615d4-616e-43a9-87ef-747844e14570-crio-socket\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.517259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.517185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/092615d4-616e-43a9-87ef-747844e14570-data-volume\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.517259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.517203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.517259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.517218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j6mj\" (UniqueName: \"kubernetes.io/projected/092615d4-616e-43a9-87ef-747844e14570-kube-api-access-8j6mj\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.517372 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.517276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.563379 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.563357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:19.617589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.617664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/092615d4-616e-43a9-87ef-747844e14570-crio-socket\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.617664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/092615d4-616e-43a9-87ef-747844e14570-data-volume\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.617749 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:19.617662 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 22:01:19.617749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.617749 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:19.617717 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.117702498 +0000 UTC m=+65.789200450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : configmap "kube-rbac-proxy" not found Sep 18 22:01:19.617749 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:19.617728 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 22:01:19.617749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6mj\" (UniqueName: \"kubernetes.io/projected/092615d4-616e-43a9-87ef-747844e14570-kube-api-access-8j6mj\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.618025 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.617744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/092615d4-616e-43a9-87ef-747844e14570-crio-socket\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.618025 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:19.617773 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:20.117758075 +0000 UTC m=+65.789256037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : secret "insights-runtime-extractor-tls" not found Sep 18 22:01:19.618130 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.618086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/092615d4-616e-43a9-87ef-747844e14570-data-volume\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.627447 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.627427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6mj\" (UniqueName: \"kubernetes.io/projected/092615d4-616e-43a9-87ef-747844e14570-kube-api-access-8j6mj\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:19.703906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.703871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:19.781417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:19.781395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:20.122209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.122183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:20.122338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.122222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:20.122338 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.122284 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 22:01:20.122338 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.122314 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 22:01:20.122338 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.122332 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:21.122317931 +0000 UTC m=+66.793815892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : configmap "kube-rbac-proxy" not found Sep 18 22:01:20.122488 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.122359 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:21.122347334 +0000 UTC m=+66.793845283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : secret "insights-runtime-extractor-tls" not found Sep 18 22:01:20.525798 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.525750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:01:20.528489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.528474 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 22:01:20.536670 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.536651 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 22:01:20.536730 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.536698 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs podName:6b80a67d-582a-495f-9985-2b0bbbacde46 nodeName:}" failed. No retries permitted until 2026-09-18 22:02:24.536684616 +0000 UTC m=+130.208182563 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs") pod "network-metrics-daemon-v98jj" (UID: "6b80a67d-582a-495f-9985-2b0bbbacde46") : secret "metrics-daemon-secret" not found Sep 18 22:01:20.626400 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.626377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:01:20.626483 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.626473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:01:20.628768 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.628751 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 22:01:20.628835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.628773 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 22:01:20.639760 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.639742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d0405393-1b0b-446e-b958-231c5d2b0799-original-pull-secret\") pod \"global-pull-secret-syncer-dmkfl\" (UID: \"d0405393-1b0b-446e-b958-231c5d2b0799\") " pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:01:20.639994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.639981 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 22:01:20.650186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.650166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb472\" (UniqueName: \"kubernetes.io/projected/8dd12078-56f8-4276-8e34-cce23a44e7e4-kube-api-access-cb472\") pod \"network-check-target-stszb\" (UID: \"8dd12078-56f8-4276-8e34-cce23a44e7e4\") " pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:01:20.653873 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.653847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dmkfl" Sep 18 22:01:20.664408 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.664389 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8l8bl\"" Sep 18 22:01:20.671177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.671157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:01:20.702180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.702154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:20.929477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.929408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:20.929477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.929452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:20.929642 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.929568 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:20.929642 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.929568 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 22:01:20.929642 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.929633 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:28.929614709 +0000 UTC m=+74.601112670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:20.929753 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:20.929650 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls podName:9af01b02-674e-45bf-a307-d6d96a6ead9b nodeName:}" failed. No retries permitted until 2026-09-18 22:01:28.929642143 +0000 UTC m=+74.601140104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-nnfn8" (UID: "9af01b02-674e-45bf-a307-d6d96a6ead9b") : secret "samples-operator-tls" not found Sep 18 22:01:20.941423 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.941401 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dmkfl"] Sep 18 22:01:20.944783 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:20.944759 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0405393_1b0b_446e_b958_231c5d2b0799.slice/crio-f3c5393337631375d7a99285dc6b082019ebf058bfc3eafb7a0babec39214dfa WatchSource:0}: Error finding container f3c5393337631375d7a99285dc6b082019ebf058bfc3eafb7a0babec39214dfa: Status 404 returned error can't find the container with id f3c5393337631375d7a99285dc6b082019ebf058bfc3eafb7a0babec39214dfa Sep 18 22:01:20.962441 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:20.962419 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stszb"] Sep 18 22:01:20.964347 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:20.964321 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd12078_56f8_4276_8e34_cce23a44e7e4.slice/crio-8d341a555b4188d9ea7295b06d079c9356be8711e556361906965bbd52ae3848 WatchSource:0}: Error finding container 8d341a555b4188d9ea7295b06d079c9356be8711e556361906965bbd52ae3848: Status 404 returned error can't find the container with id 8d341a555b4188d9ea7295b06d079c9356be8711e556361906965bbd52ae3848 Sep 18 22:01:21.098196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:21.098167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stszb" event={"ID":"8dd12078-56f8-4276-8e34-cce23a44e7e4","Type":"ContainerStarted","Data":"8d341a555b4188d9ea7295b06d079c9356be8711e556361906965bbd52ae3848"} Sep 18 22:01:21.099281 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:21.099260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dmkfl" event={"ID":"d0405393-1b0b-446e-b958-231c5d2b0799","Type":"ContainerStarted","Data":"f3c5393337631375d7a99285dc6b082019ebf058bfc3eafb7a0babec39214dfa"} Sep 18 22:01:21.130770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:21.130749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:21.130874 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:21.130803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:21.130931 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:21.130911 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 22:01:21.130931 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:21.130909 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 22:01:21.131027 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:21.130955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:23.130942799 +0000 UTC m=+68.802440752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : configmap "kube-rbac-proxy" not found Sep 18 22:01:21.131027 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:21.130978 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:23.130960439 +0000 UTC m=+68.802458391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : secret "insights-runtime-extractor-tls" not found Sep 18 22:01:22.038618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:22.038583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:22.038631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:22.038746 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:22.038761 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5b64f97c-rvj8p: secret "image-registry-tls" not found Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:22.038813 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:22.038834 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls podName:a7badbe7-0f56-4946-9961-f9e3900a560f nodeName:}" failed. No retries permitted until 2026-09-18 22:01:54.03881542 +0000 UTC m=+99.710313375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls") pod "image-registry-5c5b64f97c-rvj8p" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f") : secret "image-registry-tls" not found Sep 18 22:01:22.039028 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:22.038951 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls podName:34307b55-2153-4ea8-914a-9c5fbffe1deb nodeName:}" failed. No retries permitted until 2026-09-18 22:01:54.038924125 +0000 UTC m=+99.710422076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls") pod "dns-default-bdkwr" (UID: "34307b55-2153-4ea8-914a-9c5fbffe1deb") : secret "dns-default-metrics-tls" not found Sep 18 22:01:22.253471 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:22.253431 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6wxg_33ac72fd-379b-4bc4-9fec-7942cf4ee26b/dns-node-resolver/0.log" Sep 18 22:01:23.048565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:23.048518 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qltq_c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92/node-ca/0.log" Sep 18 22:01:23.147738 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:23.147699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:23.147934 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:23.147767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:23.147934 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:23.147872 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 22:01:23.148054 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:23.147955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:27.147935189 +0000 UTC m=+72.819433136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : configmap "kube-rbac-proxy" not found Sep 18 22:01:23.148353 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:23.148333 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 22:01:23.148418 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:23.148387 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:27.148376313 +0000 UTC m=+72.819874260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : secret "insights-runtime-extractor-tls" not found Sep 18 22:01:24.051988 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.051950 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d6zt4_6e137694-e4b9-4aed-8de3-32e7f4984392/serve-healthcheck-canary/0.log" Sep 18 22:01:24.079867 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.079776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.178169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.178137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.289077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.289036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.389114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.389037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.487131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.487095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.726680 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.726628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:24.922693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:24.922658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:25.198110 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:25.198045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:25.926707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:25.926679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:26.112840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.112813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dmkfl" event={"ID":"d0405393-1b0b-446e-b958-231c5d2b0799","Type":"ContainerStarted","Data":"428f5790918575c94543c907f4cca9675471c757648af26b5d54b1444f608401"} Sep 18 22:01:26.114441 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.114414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stszb" event={"ID":"8dd12078-56f8-4276-8e34-cce23a44e7e4","Type":"ContainerStarted","Data":"895dd42f6d0a6a5569ba5233c531f711cdb812c2593181584b5f1e55c3f44ebe"} Sep 18 22:01:26.115278 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.114867 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:01:26.142302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.142258 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-stszb" podStartSLOduration=66.151772664 podStartE2EDuration="1m11.14224451s" totalImagesPullingTime="4.990471846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:26.141730686 +0000 UTC m=+71.813228656" watchObservedRunningTime="2026-09-18 22:01:26.14224451 +0000 UTC m=+71.813742477" Sep 18 22:01:26.341457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.341429 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:26.577496 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.577470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:26.683128 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.683098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:26.783778 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:26.783751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:27.180588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:27.180519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:27.180588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:27.180557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:27.180779 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:27.180666 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 22:01:27.180779 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:27.180713 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 22:01:27.180779 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:27.180741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:35.180719442 +0000 UTC m=+80.852217400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : secret "insights-runtime-extractor-tls" not found Sep 18 22:01:27.180779 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:27.180758 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm podName:092615d4-616e-43a9-87ef-747844e14570 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:35.180747011 +0000 UTC m=+80.852244982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2xnq2" (UID: "092615d4-616e-43a9-87ef-747844e14570") : configmap "kube-rbac-proxy" not found Sep 18 22:01:27.193417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:27.193392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:27.527043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:27.527007 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:27.639346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:27.639321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.147463 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.147429 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.329983 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.329955 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.512817 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.512776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.716197 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.716164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.814049 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.813978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:28.995075 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.995047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:28.995189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.995088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:28.995189 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:28.995177 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:28.995256 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:01:28.995228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls podName:96343875-434b-4192-8130-794208b79619 nodeName:}" failed. No retries permitted until 2026-09-18 22:01:44.995215611 +0000 UTC m=+90.666713572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-lw8ph" (UID: "96343875-434b-4192-8130-794208b79619") : secret "cluster-monitoring-operator-tls" not found Sep 18 22:01:28.997606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:28.997578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9af01b02-674e-45bf-a307-d6d96a6ead9b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-nnfn8\" (UID: \"9af01b02-674e-45bf-a307-d6d96a6ead9b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:29.115865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.115818 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b4tfr\"" Sep 18 22:01:29.122437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.122405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:29.124374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.124361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" Sep 18 22:01:29.392748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.392664 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dmkfl" podStartSLOduration=69.370921331 podStartE2EDuration="1m14.392646001s" totalImagesPullingTime="5.02172467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:27.160191338 +0000 UTC m=+72.831689310" watchObservedRunningTime="2026-09-18 22:01:29.392646001 +0000 UTC m=+75.064143980" Sep 18 22:01:29.393298 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.393280 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8"] Sep 18 22:01:29.518006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:29.517984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:30.116171 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.116145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:30.123895 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.123867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" event={"ID":"9af01b02-674e-45bf-a307-d6d96a6ead9b","Type":"ContainerStarted","Data":"468a34a8cd600447925b654a76e05c329c9ce38b621bc332c7697d59fe1c507b"} Sep 18 22:01:30.372086 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.372012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:30.528209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.528176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:30.716655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.716628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:30.962317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:30.962285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:31.116062 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:31.115992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:31.550579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:31.550549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:32.122400 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.122366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:32.135964 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.135935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" event={"ID":"9af01b02-674e-45bf-a307-d6d96a6ead9b","Type":"ContainerStarted","Data":"3431e694bbcfbb984e8384d4cec0c60674f4b1aa0d66719247256485c1d51b83"} Sep 18 22:01:32.136089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.136005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" event={"ID":"9af01b02-674e-45bf-a307-d6d96a6ead9b","Type":"ContainerStarted","Data":"45c2fdc621ffc1448222e63834c23ac191684bdcab1b2896b64b497bb0bc729a"} Sep 18 22:01:32.161199 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.161160 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-nnfn8" podStartSLOduration=16.985751347 podStartE2EDuration="19.161146805s" totalImagesPullingTime="2.175395458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:32.160979066 +0000 UTC m=+77.832477035" watchObservedRunningTime="2026-09-18 22:01:32.161146805 +0000 UTC m=+77.832644775" Sep 18 22:01:32.419075 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.419000 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:32.780417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.780389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:32.976239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:32.976218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:33.319063 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:33.319024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:33.515699 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:33.515668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:34.127390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:34.127358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:34.330282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:34.330249 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:34.526415 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:34.526387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.126673 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.126647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.227709 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.227686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.240791 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.240771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:35.240850 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.240821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:35.241398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.241372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/092615d4-616e-43a9-87ef-747844e14570-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:35.243279 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.243259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/092615d4-616e-43a9-87ef-747844e14570-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2xnq2\" (UID: \"092615d4-616e-43a9-87ef-747844e14570\") " pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:35.399816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.399742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2xnq2" Sep 18 22:01:35.400201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.400168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.567257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.567223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.639770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.639745 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.680639 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.680586 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2xnq2"] Sep 18 22:01:35.683711 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:35.683681 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092615d4_616e_43a9_87ef_747844e14570.slice/crio-8ce9a50af3d6f4c0348f04161559a5d30040843e765113ce35751e2299bc4b0f WatchSource:0}: Error finding container 8ce9a50af3d6f4c0348f04161559a5d30040843e765113ce35751e2299bc4b0f: Status 404 returned error can't find the container with id 8ce9a50af3d6f4c0348f04161559a5d30040843e765113ce35751e2299bc4b0f Sep 18 22:01:35.762980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.762953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:35.864269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:35.864244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:36.146991 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.146881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2xnq2" event={"ID":"092615d4-616e-43a9-87ef-747844e14570","Type":"ContainerStarted","Data":"cbb51e8394a09bf1094429c2f317e4adecf2f3058ed383b92e541f09647b3837"} Sep 18 22:01:36.146991 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.146969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2xnq2" event={"ID":"092615d4-616e-43a9-87ef-747844e14570","Type":"ContainerStarted","Data":"8ce9a50af3d6f4c0348f04161559a5d30040843e765113ce35751e2299bc4b0f"} Sep 18 22:01:36.398282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.398193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:36.511510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.511484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:36.615488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.615463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:36.787555 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.787525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:36.964694 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:36.964662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:37.130039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:37.129978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:37.152627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:37.152602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2xnq2" event={"ID":"092615d4-616e-43a9-87ef-747844e14570","Type":"ContainerStarted","Data":"a1014a56965696367631bd998401dff43fd3c027011759745079c66687703b1f"} Sep 18 22:01:38.158014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:38.157985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2xnq2" event={"ID":"092615d4-616e-43a9-87ef-747844e14570","Type":"ContainerStarted","Data":"4578431801c13c8002f14f803b0945d85c55ee04cb90bf3bbf7def55256cd26f"} Sep 18 22:01:38.177520 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:38.177478 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2xnq2" podStartSLOduration=16.923260394 podStartE2EDuration="19.17746512s" totalImagesPullingTime="2.254204726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:01:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:38.176475798 +0000 UTC m=+83.847973797" watchObservedRunningTime="2026-09-18 22:01:38.17746512 +0000 UTC m=+83.848963089" Sep 18 22:01:40.788004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:40.787971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:40.853238 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:40.853198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:42.017405 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:42.017356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:42.088143 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:42.088113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:44.086577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.086542 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b"] Sep 18 22:01:44.089622 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.089608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.093378 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.093355 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jblm"] Sep 18 22:01:44.094231 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.094209 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 22:01:44.094376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.094267 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 18 22:01:44.094376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.094267 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 22:01:44.094376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.094267 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 22:01:44.096014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.095998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.134701 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.134677 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 18 22:01:44.135010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.134996 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7jrtb\"" Sep 18 22:01:44.135174 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.135164 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 18 22:01:44.144660 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.144640 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b"] Sep 18 22:01:44.195926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.195906 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jblm"] Sep 18 22:01:44.201277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.201258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e99cf92a-8040-4f12-8ba4-4b8801e1da30-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.201365 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.201289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgv6z\" (UniqueName: \"kubernetes.io/projected/c38f4046-8527-4c25-a382-97de18f4f166-kube-api-access-wgv6z\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.201365 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.201351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e99cf92a-8040-4f12-8ba4-4b8801e1da30-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.201477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.201391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38f4046-8527-4c25-a382-97de18f4f166-tmp\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.201477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.201424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c38f4046-8527-4c25-a382-97de18f4f166-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.302328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e99cf92a-8040-4f12-8ba4-4b8801e1da30-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.302430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgv6z\" (UniqueName: \"kubernetes.io/projected/c38f4046-8527-4c25-a382-97de18f4f166-kube-api-access-wgv6z\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.302430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e99cf92a-8040-4f12-8ba4-4b8801e1da30-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.302430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38f4046-8527-4c25-a382-97de18f4f166-tmp\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.302587 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c38f4046-8527-4c25-a382-97de18f4f166-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.302855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.302830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38f4046-8527-4c25-a382-97de18f4f166-tmp\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.303052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.303035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e99cf92a-8040-4f12-8ba4-4b8801e1da30-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.305043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.305015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e99cf92a-8040-4f12-8ba4-4b8801e1da30-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jblm\" (UID: \"e99cf92a-8040-4f12-8ba4-4b8801e1da30\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.305112 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.305094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c38f4046-8527-4c25-a382-97de18f4f166-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.328968 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.328945 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-prtpp"] Sep 18 22:01:44.332075 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.332055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:01:44.335618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.335596 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 22:01:44.335712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.335622 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 22:01:44.335803 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.335739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wln6q\"" Sep 18 22:01:44.338133 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.338101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgv6z\" (UniqueName: \"kubernetes.io/projected/c38f4046-8527-4c25-a382-97de18f4f166-kube-api-access-wgv6z\") pod \"klusterlet-addon-workmgr-5d6fdfb89f-wnw5b\" (UID: \"c38f4046-8527-4c25-a382-97de18f4f166\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.371597 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.371548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-prtpp"] Sep 18 22:01:44.399632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.399613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:44.403377 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.403363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmdzr\" (UniqueName: \"kubernetes.io/projected/df3b83c5-db34-4419-bfd6-3349890b09fc-kube-api-access-cmdzr\") pod \"downloads-6594f4c75-prtpp\" (UID: \"df3b83c5-db34-4419-bfd6-3349890b09fc\") " pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:01:44.404364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.404347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" Sep 18 22:01:44.479455 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.479427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:44.504497 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.504476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmdzr\" (UniqueName: \"kubernetes.io/projected/df3b83c5-db34-4419-bfd6-3349890b09fc-kube-api-access-cmdzr\") pod \"downloads-6594f4c75-prtpp\" (UID: \"df3b83c5-db34-4419-bfd6-3349890b09fc\") " pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:01:44.519714 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.519692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmdzr\" (UniqueName: \"kubernetes.io/projected/df3b83c5-db34-4419-bfd6-3349890b09fc-kube-api-access-cmdzr\") pod \"downloads-6594f4c75-prtpp\" (UID: \"df3b83c5-db34-4419-bfd6-3349890b09fc\") " pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:01:44.601923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.601867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:44.645196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.645171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:01:44.685905 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.685879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b"] Sep 18 22:01:44.688675 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:44.688636 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38f4046_8527_4c25_a382_97de18f4f166.slice/crio-50ba3b589cdd6d6ce17b4ef404728d603735be5a5f2042abaeca0c65a331e5d4 WatchSource:0}: Error finding container 50ba3b589cdd6d6ce17b4ef404728d603735be5a5f2042abaeca0c65a331e5d4: Status 404 returned error can't find the container with id 50ba3b589cdd6d6ce17b4ef404728d603735be5a5f2042abaeca0c65a331e5d4 Sep 18 22:01:44.692361 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.692296 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jblm"] Sep 18 22:01:44.695120 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:44.695098 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99cf92a_8040_4f12_8ba4_4b8801e1da30.slice/crio-02d5c578170ba60675f9b9a6fefc7d53fce6daf1a8e1108d7df4036f2d7595bc WatchSource:0}: Error finding container 02d5c578170ba60675f9b9a6fefc7d53fce6daf1a8e1108d7df4036f2d7595bc: Status 404 returned error can't find the container with id 02d5c578170ba60675f9b9a6fefc7d53fce6daf1a8e1108d7df4036f2d7595bc Sep 18 22:01:44.716548 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.716527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:44.928753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:44.928666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-prtpp"] Sep 18 22:01:44.932199 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:44.932174 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3b83c5_db34_4419_bfd6_3349890b09fc.slice/crio-82a69bddae614cabb11acfec0fb1fdb04c6dcb6a53897c7a74d6e188ab9be792 WatchSource:0}: Error finding container 82a69bddae614cabb11acfec0fb1fdb04c6dcb6a53897c7a74d6e188ab9be792: Status 404 returned error can't find the container with id 82a69bddae614cabb11acfec0fb1fdb04c6dcb6a53897c7a74d6e188ab9be792 Sep 18 22:01:45.007748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.007719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:45.018089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.018060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96343875-434b-4192-8130-794208b79619-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-lw8ph\" (UID: \"96343875-434b-4192-8130-794208b79619\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:45.029860 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.029837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6jvbp\"" Sep 18 22:01:45.036010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.035996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" Sep 18 22:01:45.181314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.181243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" event={"ID":"e99cf92a-8040-4f12-8ba4-4b8801e1da30","Type":"ContainerStarted","Data":"02d5c578170ba60675f9b9a6fefc7d53fce6daf1a8e1108d7df4036f2d7595bc"} Sep 18 22:01:45.182488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.182467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-prtpp" event={"ID":"df3b83c5-db34-4419-bfd6-3349890b09fc","Type":"ContainerStarted","Data":"82a69bddae614cabb11acfec0fb1fdb04c6dcb6a53897c7a74d6e188ab9be792"} Sep 18 22:01:45.183588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.183567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" event={"ID":"c38f4046-8527-4c25-a382-97de18f4f166","Type":"ContainerStarted","Data":"50ba3b589cdd6d6ce17b4ef404728d603735be5a5f2042abaeca0c65a331e5d4"} Sep 18 22:01:45.310932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:45.310907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph"] Sep 18 22:01:45.313350 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:45.313319 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96343875_434b_4192_8130_794208b79619.slice/crio-2c23ee72f0145e841234296e470bbb38c6abd7654fd05ea1e825a7d58c0d3b22 WatchSource:0}: Error finding container 2c23ee72f0145e841234296e470bbb38c6abd7654fd05ea1e825a7d58c0d3b22: Status 404 returned error can't find the container with id 2c23ee72f0145e841234296e470bbb38c6abd7654fd05ea1e825a7d58c0d3b22 Sep 18 22:01:46.191202 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:46.191125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" event={"ID":"96343875-434b-4192-8130-794208b79619","Type":"ContainerStarted","Data":"2c23ee72f0145e841234296e470bbb38c6abd7654fd05ea1e825a7d58c0d3b22"} Sep 18 22:01:47.199940 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:47.199900 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" event={"ID":"e99cf92a-8040-4f12-8ba4-4b8801e1da30","Type":"ContainerStarted","Data":"168b66031f498b0efca0c01a14e21ee7312315594e3221116d9bb07299114e16"} Sep 18 22:01:47.218279 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:47.217807 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jblm" podStartSLOduration=1.5020345590000002 podStartE2EDuration="3.217770299s" totalImagesPullingTime="1.71573574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:47.216822575 +0000 UTC m=+92.888320546" watchObservedRunningTime="2026-09-18 22:01:47.217770299 +0000 UTC m=+92.889268270" Sep 18 22:01:47.382935 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:47.382900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:47.494875 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:47.494782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:48.791481 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:48.791445 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:48.866529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:48.866494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:49.387501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.387469 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:01:49.391057 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.391028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.393910 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.393881 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 22:01:49.394138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.394115 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 22:01:49.394844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.394817 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 22:01:49.394980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.394952 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 22:01:49.395064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.395034 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 22:01:49.395368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.395170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x59bp\"" Sep 18 22:01:49.400039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.399681 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:01:49.543689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7qt\" (UniqueName: \"kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.543863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.543863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.543863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.544096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.544096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.543899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.644921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.644863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.644921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.644905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.644960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7qt\" (UniqueName: \"kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.644984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.645008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.645059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645820 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.645776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.645948 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.645919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.646012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.645965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.647949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.647910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.648545 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.648523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.657964 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.657946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7qt\" (UniqueName: \"kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt\") pod \"console-b4f8688c8-czv2p\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.702909 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.702876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:49.980029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:49.979995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:50.009588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.009554 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:01:50.013700 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:50.013663 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2df326_6bef_45d0_a615_fc803b0dc1b9.slice/crio-e65ee6029c5f271527b81526acc58d5e7fd5125d5b53fe3273577a2b0c8a5aa1 WatchSource:0}: Error finding container e65ee6029c5f271527b81526acc58d5e7fd5125d5b53fe3273577a2b0c8a5aa1: Status 404 returned error can't find the container with id e65ee6029c5f271527b81526acc58d5e7fd5125d5b53fe3273577a2b0c8a5aa1 Sep 18 22:01:50.086923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.086859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:50.192410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.192373 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf"] Sep 18 22:01:50.196876 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.196854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:50.197141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.197118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:50.199406 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.199382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 22:01:50.199509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.199486 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2vwfb\"" Sep 18 22:01:50.206852 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.206832 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf"] Sep 18 22:01:50.212599 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.212548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4f8688c8-czv2p" event={"ID":"9e2df326-6bef-45d0-a615-fc803b0dc1b9","Type":"ContainerStarted","Data":"e65ee6029c5f271527b81526acc58d5e7fd5125d5b53fe3273577a2b0c8a5aa1"} Sep 18 22:01:50.214353 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.214334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" event={"ID":"c38f4046-8527-4c25-a382-97de18f4f166","Type":"ContainerStarted","Data":"8805f1444335285a4da594932b0f65947913241dcad2f71e879cf344a650fcc5"} Sep 18 22:01:50.214542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.214522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:50.216165 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.216147 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" Sep 18 22:01:50.216274 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.216260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" event={"ID":"96343875-434b-4192-8130-794208b79619","Type":"ContainerStarted","Data":"b2cfa3dfeed1b125afaada99686ea8e7f290bb2f7930862c99355ee16280c5b9"} Sep 18 22:01:50.247897 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.247783 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-lw8ph" podStartSLOduration=32.97421467 podStartE2EDuration="37.247763842s" totalImagesPullingTime="4.273549172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:50.246908272 +0000 UTC m=+95.918406280" watchObservedRunningTime="2026-09-18 22:01:50.247763842 +0000 UTC m=+95.919261814" Sep 18 22:01:50.266532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.266482 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d6fdfb89f-wnw5b" podStartSLOduration=1.350060258 podStartE2EDuration="6.266467144s" totalImagesPullingTime="4.916406886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:50.265923545 +0000 UTC m=+95.937421539" watchObservedRunningTime="2026-09-18 22:01:50.266467144 +0000 UTC m=+95.937965115" Sep 18 22:01:50.351915 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.351525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fbb370a-8423-401d-ad24-2b032c1db630-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9fqgf\" (UID: \"5fbb370a-8423-401d-ad24-2b032c1db630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:50.452215 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.452184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fbb370a-8423-401d-ad24-2b032c1db630-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9fqgf\" (UID: \"5fbb370a-8423-401d-ad24-2b032c1db630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:50.454859 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.454838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fbb370a-8423-401d-ad24-2b032c1db630-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9fqgf\" (UID: \"5fbb370a-8423-401d-ad24-2b032c1db630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:50.486269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.486241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:50.508462 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.508314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:50.788570 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:50.788549 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf"] Sep 18 22:01:51.234224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:51.234093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" event={"ID":"5fbb370a-8423-401d-ad24-2b032c1db630","Type":"ContainerStarted","Data":"6fe8c3ff5c46a76d8c83e883ee7d5481ef20d9d4095caf08011e231ee5c1d079"} Sep 18 22:01:51.995649 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:51.995602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:52.071042 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:52.071005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:53.149920 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:53.149875 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:53.249828 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:53.249773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:53.408803 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:53.408661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:53.488372 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:53.488316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:54.081541 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.081502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:01:54.081699 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.081548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:54.084529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.084479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"image-registry-5c5b64f97c-rvj8p\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:54.084984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.084962 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34307b55-2153-4ea8-914a-9c5fbffe1deb-metrics-tls\") pod \"dns-default-bdkwr\" (UID: \"34307b55-2153-4ea8-914a-9c5fbffe1deb\") " pod="openshift-dns/dns-default-bdkwr" Sep 18 22:01:54.361002 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.360922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m9bdj\"" Sep 18 22:01:54.365989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.365967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:54.372221 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.372060 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdhvj\"" Sep 18 22:01:54.380689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.380667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdkwr" Sep 18 22:01:54.998941 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:54.998908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:55.064640 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.064595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:55.244992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.244942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" event={"ID":"5fbb370a-8423-401d-ad24-2b032c1db630","Type":"ContainerStarted","Data":"5b47136212c9fe70e8d7d8c5c42f5c23665c88d0df94ed1f19f3e2c8988f8f8e"} Sep 18 22:01:55.399266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.399221 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdkwr"] Sep 18 22:01:55.401398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.401378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:01:55.472334 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:55.472303 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34307b55_2153_4ea8_914a_9c5fbffe1deb.slice/crio-642a654c33dbff3f23b8b253947ed65aaefa2c6c82937b85995fad6bc7116679 WatchSource:0}: Error finding container 642a654c33dbff3f23b8b253947ed65aaefa2c6c82937b85995fad6bc7116679: Status 404 returned error can't find the container with id 642a654c33dbff3f23b8b253947ed65aaefa2c6c82937b85995fad6bc7116679 Sep 18 22:01:55.472891 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:01:55.472856 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7badbe7_0f56_4946_9961_f9e3900a560f.slice/crio-e68e19c4362ea32a6ad7819ac5cb65197cf37554064667d53e4b8c8ce8e8ffcc WatchSource:0}: Error finding container e68e19c4362ea32a6ad7819ac5cb65197cf37554064667d53e4b8c8ce8e8ffcc: Status 404 returned error can't find the container with id e68e19c4362ea32a6ad7819ac5cb65197cf37554064667d53e4b8c8ce8e8ffcc Sep 18 22:01:55.501435 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.501409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:55.620647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:55.620620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:56.251484 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.251386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdkwr" event={"ID":"34307b55-2153-4ea8-914a-9c5fbffe1deb","Type":"ContainerStarted","Data":"642a654c33dbff3f23b8b253947ed65aaefa2c6c82937b85995fad6bc7116679"} Sep 18 22:01:56.253755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.253694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4f8688c8-czv2p" event={"ID":"9e2df326-6bef-45d0-a615-fc803b0dc1b9","Type":"ContainerStarted","Data":"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d"} Sep 18 22:01:56.255890 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.255857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" event={"ID":"a7badbe7-0f56-4946-9961-f9e3900a560f","Type":"ContainerStarted","Data":"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7"} Sep 18 22:01:56.256000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.255919 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" event={"ID":"a7badbe7-0f56-4946-9961-f9e3900a560f","Type":"ContainerStarted","Data":"e68e19c4362ea32a6ad7819ac5cb65197cf37554064667d53e4b8c8ce8e8ffcc"} Sep 18 22:01:56.256147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.256073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:56.256902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.256858 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:01:56.261959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.261916 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" Sep 18 22:01:56.293259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.292907 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b4f8688c8-czv2p" podStartSLOduration=1.798458905 podStartE2EDuration="7.292888188s" totalImagesPullingTime="5.494429283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:56.291761781 +0000 UTC m=+101.963259752" watchObservedRunningTime="2026-09-18 22:01:56.292888188 +0000 UTC m=+101.964386162" Sep 18 22:01:56.331352 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.331288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9fqgf" podStartSLOduration=2.01291744 podStartE2EDuration="6.331270226s" totalImagesPullingTime="4.318352786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:56.330230799 +0000 UTC m=+102.001728770" watchObservedRunningTime="2026-09-18 22:01:56.331270226 +0000 UTC m=+102.002768194" Sep 18 22:01:56.367461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:56.367405 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" podStartSLOduration=100.367385642 podStartE2EDuration="1m40.367385642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:56.367091521 +0000 UTC m=+102.038589492" watchObservedRunningTime="2026-09-18 22:01:56.367385642 +0000 UTC m=+102.038883614" Sep 18 22:01:57.228540 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.228503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:57.268569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.268543 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf"] Sep 18 22:01:57.293702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.293677 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf"] Sep 18 22:01:57.293918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.293815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.296424 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.296391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 18 22:01:57.296545 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.296475 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 18 22:01:57.297334 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.297314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 22:01:57.297334 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.297333 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9468s\"" Sep 18 22:01:57.317763 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.317743 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:57.412864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.412300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.412864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.412536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.412864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.412574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66c566bf-133c-414f-ac85-81c8dd493eb8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.412864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.412682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9th\" (UniqueName: \"kubernetes.io/projected/66c566bf-133c-414f-ac85-81c8dd493eb8-kube-api-access-hh9th\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.507885 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.507818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:01:57.513757 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.513731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.513911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.513778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.513911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.513825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66c566bf-133c-414f-ac85-81c8dd493eb8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.513911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.513867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9th\" (UniqueName: \"kubernetes.io/projected/66c566bf-133c-414f-ac85-81c8dd493eb8-kube-api-access-hh9th\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.514638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.514615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66c566bf-133c-414f-ac85-81c8dd493eb8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.516772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.516748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.516867 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.516779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66c566bf-133c-414f-ac85-81c8dd493eb8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.523383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.523350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9th\" (UniqueName: \"kubernetes.io/projected/66c566bf-133c-414f-ac85-81c8dd493eb8-kube-api-access-hh9th\") pod \"prometheus-operator-7bd5d56d6-wwpsf\" (UID: \"66c566bf-133c-414f-ac85-81c8dd493eb8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:57.605228 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:57.605187 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" Sep 18 22:01:58.121554 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:58.121524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-stszb" Sep 18 22:01:59.703594 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:59.703555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:59.704089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:59.703624 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:01:59.708979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:01:59.708950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:02:00.276632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:00.276599 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:02:00.430625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:00.430590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:00.529556 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:00.529478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:00.755089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:00.755039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:00.856210 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:00.856118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:03.853242 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:03.853207 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:02:03.976262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:03.976214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:04.077108 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:04.077077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:04.179972 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:04.179902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:04.740135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:04.740111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf"] Sep 18 22:02:04.742019 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:04.741995 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c566bf_133c_414f_ac85_81c8dd493eb8.slice/crio-f84b7281eff2671e18226413c90e2094a5fe3397183f69b3743a429c52af8e57 WatchSource:0}: Error finding container f84b7281eff2671e18226413c90e2094a5fe3397183f69b3743a429c52af8e57: Status 404 returned error can't find the container with id f84b7281eff2671e18226413c90e2094a5fe3397183f69b3743a429c52af8e57 Sep 18 22:02:05.292036 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.291994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdkwr" event={"ID":"34307b55-2153-4ea8-914a-9c5fbffe1deb","Type":"ContainerStarted","Data":"415b0a97c5659d848cd52e301638c73daad5597ed73d642fe64218ca1b7c76a9"} Sep 18 22:02:05.292500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.292061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdkwr" event={"ID":"34307b55-2153-4ea8-914a-9c5fbffe1deb","Type":"ContainerStarted","Data":"be1736a8c18c3cb2651a16de6f7acd28b3be6218e46ee5e5e4f3b1cee9f1f79b"} Sep 18 22:02:05.292500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.292108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bdkwr" Sep 18 22:02:05.293726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.293691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" event={"ID":"66c566bf-133c-414f-ac85-81c8dd493eb8","Type":"ContainerStarted","Data":"f84b7281eff2671e18226413c90e2094a5fe3397183f69b3743a429c52af8e57"} Sep 18 22:02:05.295574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.295546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-prtpp" event={"ID":"df3b83c5-db34-4419-bfd6-3349890b09fc","Type":"ContainerStarted","Data":"67c7ef88e2b4f46b1e69b73e05cc8023fcfc590815fcd39a7c6b6385df5caebb"} Sep 18 22:02:05.296007 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.295983 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:02:05.312411 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.312386 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-prtpp" Sep 18 22:02:05.312527 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.312372 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bdkwr" podStartSLOduration=66.33154705 podStartE2EDuration="1m15.312355153s" totalImagesPullingTime="8.980808103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:05.312036233 +0000 UTC m=+110.983534204" watchObservedRunningTime="2026-09-18 22:02:05.312355153 +0000 UTC m=+110.983853124" Sep 18 22:02:05.330045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:05.329999 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-prtpp" podStartSLOduration=1.775078468 podStartE2EDuration="21.32998283s" totalImagesPullingTime="19.554904362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:05.329447212 +0000 UTC m=+111.000945183" watchObservedRunningTime="2026-09-18 22:02:05.32998283 +0000 UTC m=+111.001480801" Sep 18 22:02:06.864432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:06.864399 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:02:06.997995 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:06.997965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:07.071969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:07.071897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:07.207697 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:07.207664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:07.306864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:07.306823 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" event={"ID":"66c566bf-133c-414f-ac85-81c8dd493eb8","Type":"ContainerStarted","Data":"b6d3508865dc8720e6a46522772f994d1d2af04d21d49e236e6067eb780930d6"} Sep 18 22:02:07.307038 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:07.306885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" event={"ID":"66c566bf-133c-414f-ac85-81c8dd493eb8","Type":"ContainerStarted","Data":"4907f8603bc77e896234bd3842dde4c45662d0322f7e7bf03fe4489d4b00b3fc"} Sep 18 22:02:07.326682 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:07.326595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-wwpsf" podStartSLOduration=8.271647982 podStartE2EDuration="10.326576203s" totalImagesPullingTime="2.054928221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:07.325184915 +0000 UTC m=+112.996682890" watchObservedRunningTime="2026-09-18 22:02:07.326576203 +0000 UTC m=+112.998074174" Sep 18 22:02:09.386663 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.386629 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:09.685429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.685388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:09.694556 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.694524 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cfk2s"] Sep 18 22:02:09.723873 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.723848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.726115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.726091 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 22:02:09.726243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.726127 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 22:02:09.726243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.726154 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vz5jd\"" Sep 18 22:02:09.726463 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.726447 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 22:02:09.776432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.776406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:09.820399 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-sys\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-wtmp\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-textfile\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-metrics-client-ca\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jt9\" (UniqueName: \"kubernetes.io/projected/895fe44e-0d74-496e-98ce-0e4579ea112f-kube-api-access-z5jt9\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-tls\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.820889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.820772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-root\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.876962 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.876933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:09.921736 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-textfile\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.921922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-metrics-client-ca\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.921922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.921922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.921922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jt9\" (UniqueName: \"kubernetes.io/projected/895fe44e-0d74-496e-98ce-0e4579ea112f-kube-api-access-z5jt9\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.921922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-tls\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922192 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.921964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-root\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922192 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-textfile\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922294 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-root\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922388 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-sys\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-metrics-client-ca\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-sys\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922617 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-wtmp\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922738 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.922832 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.922753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-wtmp\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.925166 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.925142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.925253 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.925207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895fe44e-0d74-496e-98ce-0e4579ea112f-node-exporter-tls\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.929980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.929959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jt9\" (UniqueName: \"kubernetes.io/projected/895fe44e-0d74-496e-98ce-0e4579ea112f-kube-api-access-z5jt9\") pod \"node-exporter-cfk2s\" (UID: \"895fe44e-0d74-496e-98ce-0e4579ea112f\") " pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:09.977781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:09.977719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.034992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.034964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfk2s" Sep 18 22:02:10.064664 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:10.064623 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895fe44e_0d74_496e_98ce_0e4579ea112f.slice/crio-ee6802ae1996498340fc392ce9efe8816cfe824203b61addfbca770a6622f779 WatchSource:0}: Error finding container ee6802ae1996498340fc392ce9efe8816cfe824203b61addfbca770a6622f779: Status 404 returned error can't find the container with id ee6802ae1996498340fc392ce9efe8816cfe824203b61addfbca770a6622f779 Sep 18 22:02:10.077656 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.077631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.179034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.179006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.321139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.321058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfk2s" event={"ID":"895fe44e-0d74-496e-98ce-0e4579ea112f","Type":"ContainerStarted","Data":"ee6802ae1996498340fc392ce9efe8816cfe824203b61addfbca770a6622f779"} Sep 18 22:02:10.394807 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.394759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.627516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.627441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.726587 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.726552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:10.807468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:10.806470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:12.034746 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.034680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:12.133879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.133845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:12.331961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.331886 2577 generic.go:356] "Generic (PLEG): container finished" podID="895fe44e-0d74-496e-98ce-0e4579ea112f" containerID="bc08e8234a95094c78f6c5ae88b3a1649a1b3829ee813fc9f1b824abdec51e43" exitCode=0 Sep 18 22:02:12.332119 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.331981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfk2s" event={"ID":"895fe44e-0d74-496e-98ce-0e4579ea112f","Type":"ContainerDied","Data":"bc08e8234a95094c78f6c5ae88b3a1649a1b3829ee813fc9f1b824abdec51e43"} Sep 18 22:02:12.740589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.740556 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b86897f47-j56jw"] Sep 18 22:02:12.758296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.758269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.762022 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.761847 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 18 22:02:12.762437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.762247 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5xbr6\"" Sep 18 22:02:12.762437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.762373 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 18 22:02:12.767156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.767130 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 18 22:02:12.767331 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.767289 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3620h633l3rbf\"" Sep 18 22:02:12.767401 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.767382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 18 22:02:12.767401 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.767395 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 18 22:02:12.771021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.769251 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b86897f47-j56jw"] Sep 18 22:02:12.850105 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73b36d8-4f5c-4159-b744-220418518108-metrics-client-ca\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwpn\" (UniqueName: \"kubernetes.io/projected/a73b36d8-4f5c-4159-b744-220418518108-kube-api-access-xgwpn\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-grpc-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.850579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.850434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.852267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.852239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:12.951047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-grpc-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73b36d8-4f5c-4159-b744-220418518108-metrics-client-ca\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwpn\" (UniqueName: \"kubernetes.io/projected/a73b36d8-4f5c-4159-b744-220418518108-kube-api-access-xgwpn\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.951592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.951563 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:12.952712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.952658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73b36d8-4f5c-4159-b744-220418518108-metrics-client-ca\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.954768 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.954745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-grpc-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.955071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.955048 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.955827 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.955780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-tls\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.955966 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.955842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.956410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.956384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.957688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.957668 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a73b36d8-4f5c-4159-b744-220418518108-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:12.967171 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:12.967126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwpn\" (UniqueName: \"kubernetes.io/projected/a73b36d8-4f5c-4159-b744-220418518108-kube-api-access-xgwpn\") pod \"thanos-querier-7b86897f47-j56jw\" (UID: \"a73b36d8-4f5c-4159-b744-220418518108\") " pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:13.055277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.055203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:13.078271 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.078245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:13.339252 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.339174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfk2s" event={"ID":"895fe44e-0d74-496e-98ce-0e4579ea112f","Type":"ContainerStarted","Data":"e3efbfbae3d50683d00c44a5096bb94af7b0aa20c677452e2462c7c79699124b"} Sep 18 22:02:13.339252 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.339238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfk2s" event={"ID":"895fe44e-0d74-496e-98ce-0e4579ea112f","Type":"ContainerStarted","Data":"2b26831e86c3f94136c28287f6ce748bc8cdcd0735264e5dc84b7fe12fb3a4d6"} Sep 18 22:02:13.360117 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.360064 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cfk2s" podStartSLOduration=2.577454638 podStartE2EDuration="4.360044675s" totalImagesPullingTime="1.696992822s" totalInitContainerRuntime="85.597215ms" isStatefulPod=true podCreationTimestamp="2026-09-18 22:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:13.359711617 +0000 UTC m=+119.031209587" watchObservedRunningTime="2026-09-18 22:02:13.360044675 +0000 UTC m=+119.031542645" Sep 18 22:02:13.407478 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:13.407454 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b86897f47-j56jw"] Sep 18 22:02:13.409476 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:13.409452 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73b36d8_4f5c_4159_b744_220418518108.slice/crio-72f55a79bfb9d7a8116a348f1fa42d39d18a5319a465b981365f6d2f49762bd4 WatchSource:0}: Error finding container 72f55a79bfb9d7a8116a348f1fa42d39d18a5319a465b981365f6d2f49762bd4: Status 404 returned error can't find the container with id 72f55a79bfb9d7a8116a348f1fa42d39d18a5319a465b981365f6d2f49762bd4 Sep 18 22:02:14.046393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.046361 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-774dc96fdf-k8sdr"] Sep 18 22:02:14.071620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.071585 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-774dc96fdf-k8sdr"] Sep 18 22:02:14.071986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.071692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.076817 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.076782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 18 22:02:14.077086 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.077067 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 18 22:02:14.078777 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.078499 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 18 22:02:14.078777 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.078524 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 18 22:02:14.078777 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.078668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wbbgs\"" Sep 18 22:02:14.079104 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.079087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eoqtnsodq3mqs\"" Sep 18 22:02:14.163251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-client-certs\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163360 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-metrics-server-audit-profiles\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163360 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-tls\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ddca12e-a916-47d6-adc9-358adf5d4c3c-audit-log\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-client-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bl2\" (UniqueName: \"kubernetes.io/projected/2ddca12e-a916-47d6-adc9-358adf5d4c3c-kube-api-access-n4bl2\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.163509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.163447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.263706 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.263823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-client-certs\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.263823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-metrics-server-audit-profiles\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.263823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-tls\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.263960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ddca12e-a916-47d6-adc9-358adf5d4c3c-audit-log\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.264011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.263981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-client-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.264067 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.264016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bl2\" (UniqueName: \"kubernetes.io/projected/2ddca12e-a916-47d6-adc9-358adf5d4c3c-kube-api-access-n4bl2\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.264374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.264335 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ddca12e-a916-47d6-adc9-358adf5d4c3c-audit-log\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.264452 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.264416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.264954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.264928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ddca12e-a916-47d6-adc9-358adf5d4c3c-metrics-server-audit-profiles\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.266287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.266268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-client-certs\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.266457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.266442 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-client-ca-bundle\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.266592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.266573 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ddca12e-a916-47d6-adc9-358adf5d4c3c-secret-metrics-server-tls\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.272293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.272275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bl2\" (UniqueName: \"kubernetes.io/projected/2ddca12e-a916-47d6-adc9-358adf5d4c3c-kube-api-access-n4bl2\") pod \"metrics-server-774dc96fdf-k8sdr\" (UID: \"2ddca12e-a916-47d6-adc9-358adf5d4c3c\") " pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.345181 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.345113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"72f55a79bfb9d7a8116a348f1fa42d39d18a5319a465b981365f6d2f49762bd4"} Sep 18 22:02:14.381505 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.381477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:14.410643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.410594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:14.497082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.497048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:14.597039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.596968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:14.686564 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.686539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-774dc96fdf-k8sdr"] Sep 18 22:02:14.689177 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:14.689136 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddca12e_a916_47d6_adc9_358adf5d4c3c.slice/crio-d53355645cff2f5879548d7e512e36f7d7aaf0e0fd57a6d11088c6508075f94b WatchSource:0}: Error finding container d53355645cff2f5879548d7e512e36f7d7aaf0e0fd57a6d11088c6508075f94b: Status 404 returned error can't find the container with id d53355645cff2f5879548d7e512e36f7d7aaf0e0fd57a6d11088c6508075f94b Sep 18 22:02:14.689872 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.689841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:14.820627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:14.820601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:15.302652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:15.302622 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bdkwr" Sep 18 22:02:15.351864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:15.351831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" event={"ID":"2ddca12e-a916-47d6-adc9-358adf5d4c3c","Type":"ContainerStarted","Data":"d53355645cff2f5879548d7e512e36f7d7aaf0e0fd57a6d11088c6508075f94b"} Sep 18 22:02:16.122055 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.122017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:16.124327 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.124300 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 22:02:16.145463 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.145096 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 22:02:16.145463 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.145256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.148932 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149588 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149701 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149779 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lhn5n\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149984 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 18 22:02:16.150060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.150039 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 18 22:02:16.150467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.149984 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 18 22:02:16.150467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.150123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 18 22:02:16.150467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.150370 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 18 22:02:16.150467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.150406 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 18 22:02:16.151201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.151183 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 18 22:02:16.151686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.151667 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 18 22:02:16.151764 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.151751 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 18 22:02:16.152217 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.152197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cttkboeg5o57p\"" Sep 18 22:02:16.152866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.152848 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.179523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgb2\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-kube-api-access-bhgb2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.180120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.179983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.196697 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.196672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:16.280339 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.280855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.281465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgb2\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-kube-api-access-bhgb2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.281465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.280919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.283066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.282437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.283066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.282765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.283066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.283018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.284169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.283974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.284550 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.284395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.284550 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.284512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.286142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.286115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.287164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.288167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.288702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.289158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.290578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.291196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-config\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.291688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.291636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/694053ef-b656-42ca-a5b5-acc4f65078cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.292104 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.292012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.293293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.292942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.293293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.293094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgb2\" (UniqueName: \"kubernetes.io/projected/694053ef-b656-42ca-a5b5-acc4f65078cb-kube-api-access-bhgb2\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.298177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.296054 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/694053ef-b656-42ca-a5b5-acc4f65078cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.298281 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.298241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/694053ef-b656-42ca-a5b5-acc4f65078cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"694053ef-b656-42ca-a5b5-acc4f65078cb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.359956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.359918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"8cab0604f5f93ec0bd0becf67d5ac707412a6197003dfa481dd4e4e6272975a1"} Sep 18 22:02:16.360386 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.360055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"025172c17aadbfe597d95e8d974b015b760577db8a2e9201c79af8d206f7ba75"} Sep 18 22:02:16.479522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.479483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:16.785938 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.785879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 22:02:16.870861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:16.870828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:02:17.050062 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:17.049989 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694053ef_b656_42ca_a5b5_acc4f65078cb.slice/crio-96f08b59bac4f1fb75bbcc6d6642b2eb3a91eb3692730bf634ad6eaa3166b5be WatchSource:0}: Error finding container 96f08b59bac4f1fb75bbcc6d6642b2eb3a91eb3692730bf634ad6eaa3166b5be: Status 404 returned error can't find the container with id 96f08b59bac4f1fb75bbcc6d6642b2eb3a91eb3692730bf634ad6eaa3166b5be Sep 18 22:02:17.367514 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:17.367431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"0b2667f306b2549aea0714dbedd7d35e10acf2cdc2b6783550e0bec8535ea1a3"} Sep 18 22:02:17.368923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:17.368888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"96f08b59bac4f1fb75bbcc6d6642b2eb3a91eb3692730bf634ad6eaa3166b5be"} Sep 18 22:02:18.374447 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:18.374410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" event={"ID":"2ddca12e-a916-47d6-adc9-358adf5d4c3c","Type":"ContainerStarted","Data":"f4082460aae1a2f7989476925635aab671fec29a18ee7e6b7ce34fb497f71bc4"} Sep 18 22:02:18.392506 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:18.392466 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" podStartSLOduration=1.6845844890000001 podStartE2EDuration="4.392450929s" totalImagesPullingTime="2.70786644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:18.392134052 +0000 UTC m=+124.063632023" watchObservedRunningTime="2026-09-18 22:02:18.392450929 +0000 UTC m=+124.063948899" Sep 18 22:02:19.385589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.385557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"672927e89907e71dd3eed297818b7cf5bc53b63e96973d648fee3c0daf898d72"} Sep 18 22:02:19.386160 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.385656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"2e95e5ebde3c125edcc2bfe1d81a341cc036e35f188cd7509798caf10bdb85fa"} Sep 18 22:02:19.386160 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.385713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:19.386160 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.385752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" event={"ID":"a73b36d8-4f5c-4159-b744-220418518108","Type":"ContainerStarted","Data":"a462d64b5e8300be38a594dbee51463aa7fe4fdd3b4b70ee18db0a71ff60f156"} Sep 18 22:02:19.387339 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.387316 2577 generic.go:356] "Generic (PLEG): container finished" podID="694053ef-b656-42ca-a5b5-acc4f65078cb" containerID="199830056bf59e270e1d0a86c7db3ea4e8805869e583687320741fbdad4b05f7" exitCode=0 Sep 18 22:02:19.387443 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.387399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerDied","Data":"199830056bf59e270e1d0a86c7db3ea4e8805869e583687320741fbdad4b05f7"} Sep 18 22:02:19.412906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:19.412862 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" podStartSLOduration=2.844685568 podStartE2EDuration="7.412847823s" totalImagesPullingTime="4.568162255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:02:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:19.411088597 +0000 UTC m=+125.082586566" watchObservedRunningTime="2026-09-18 22:02:19.412847823 +0000 UTC m=+125.084345793" Sep 18 22:02:21.916584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:21.916551 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:02:21.946267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:21.946235 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:02:21.946471 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:21.946280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:21.955078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:21.954868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 22:02:22.029620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.029816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.029816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.029918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.029918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.029918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.030053 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.029921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f8q\" (UniqueName: \"kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.130951 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.130917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.130967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f8q\" (UniqueName: \"kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.131869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.132061 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.131876 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.132465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.132448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.134430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.134385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.135071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.135045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.141528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.141482 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f8q\" (UniqueName: \"kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q\") pod \"console-bd4559cfc-cj5vk\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.189502 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.189468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:22.260131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.259884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:22.306375 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:22.306349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:23.012546 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.012515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:23.127969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.127816 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:02:23.133914 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.133891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:23.225757 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.225737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:23.254169 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:23.254146 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae73299b_0550_4e91_b977_2afc9f5bc7f2.slice/crio-f3dc3a316d1d1d45e49b7c5dccc9d4ae805d56366935755f5da51a2773e3f893 WatchSource:0}: Error finding container f3dc3a316d1d1d45e49b7c5dccc9d4ae805d56366935755f5da51a2773e3f893: Status 404 returned error can't find the container with id f3dc3a316d1d1d45e49b7c5dccc9d4ae805d56366935755f5da51a2773e3f893 Sep 18 22:02:23.408223 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.408194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd4559cfc-cj5vk" event={"ID":"ae73299b-0550-4e91-b977-2afc9f5bc7f2","Type":"ContainerStarted","Data":"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f"} Sep 18 22:02:23.408350 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.408246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd4559cfc-cj5vk" event={"ID":"ae73299b-0550-4e91-b977-2afc9f5bc7f2","Type":"ContainerStarted","Data":"f3dc3a316d1d1d45e49b7c5dccc9d4ae805d56366935755f5da51a2773e3f893"} Sep 18 22:02:23.410424 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:23.410402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"ad239ad836948a5b2ca5e46b363cfc85b90308323a519d13c0f688535eaa1e3a"} Sep 18 22:02:24.417996 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.417964 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"66b307accffd04ec4a9093b0b9099b25d98ba587a234d9e5c0e7761a141d246a"} Sep 18 22:02:24.418413 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.418027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"77a25580510cbc9329e1db2f776fe8e7f8f7bc1d4495367596464ee4a5a7d9b3"} Sep 18 22:02:24.418413 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.418055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"aa99e8967953cbd053f23b491f5c3d29b7cac3f281915e809db1e14e7de99596"} Sep 18 22:02:24.418413 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.418077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"d6ec263a3c88a1a0025e02acdbb06618167fa22491c08a1d8fd91bf12ccf9ed8"} Sep 18 22:02:24.418413 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.418100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"694053ef-b656-42ca-a5b5-acc4f65078cb","Type":"ContainerStarted","Data":"9ad4dcb6559eb3c15ccf723e19890e75be920bb3fedad815aca9eb0dca93c972"} Sep 18 22:02:24.456718 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.456678 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.887708924 podStartE2EDuration="8.456657444s" totalImagesPullingTime="5.542313497s" totalInitContainerRuntime="26.635023ms" isStatefulPod=false podCreationTimestamp="2026-09-18 22:02:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:24.454921862 +0000 UTC m=+130.126419833" watchObservedRunningTime="2026-09-18 22:02:24.456657444 +0000 UTC m=+130.128155419" Sep 18 22:02:24.457432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.457391 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bd4559cfc-cj5vk" podStartSLOduration=3.457378621 podStartE2EDuration="3.457378621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:23.452021578 +0000 UTC m=+129.123519541" watchObservedRunningTime="2026-09-18 22:02:24.457378621 +0000 UTC m=+130.128876592" Sep 18 22:02:24.483946 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.483917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:24.555718 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.555696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:02:24.558047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.558025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b80a67d-582a-495f-9985-2b0bbbacde46-metrics-certs\") pod \"network-metrics-daemon-v98jj\" (UID: \"6b80a67d-582a-495f-9985-2b0bbbacde46\") " pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:02:24.561332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.561313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:24.759994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.759973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27j86\"" Sep 18 22:02:24.768267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:24.768251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v98jj" Sep 18 22:02:25.071179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:25.071118 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v98jj"] Sep 18 22:02:25.073752 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:02:25.073717 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b80a67d_582a_495f_9985_2b0bbbacde46.slice/crio-9c8331bec003dae4592c1770261c64dbb370324477cfe606a7b47f79c3e560f3 WatchSource:0}: Error finding container 9c8331bec003dae4592c1770261c64dbb370324477cfe606a7b47f79c3e560f3: Status 404 returned error can't find the container with id 9c8331bec003dae4592c1770261c64dbb370324477cfe606a7b47f79c3e560f3 Sep 18 22:02:25.397611 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:25.397541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b86897f47-j56jw" Sep 18 22:02:25.423109 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:25.423084 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v98jj" event={"ID":"6b80a67d-582a-495f-9985-2b0bbbacde46","Type":"ContainerStarted","Data":"9c8331bec003dae4592c1770261c64dbb370324477cfe606a7b47f79c3e560f3"} Sep 18 22:02:26.480158 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:26.480124 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:02:27.432654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:27.432625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v98jj" event={"ID":"6b80a67d-582a-495f-9985-2b0bbbacde46","Type":"ContainerStarted","Data":"9015d5507927a5154f9698409f57d92758cec92eba1498b06cb390237c756e66"} Sep 18 22:02:27.432807 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:27.432674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v98jj" event={"ID":"6b80a67d-582a-495f-9985-2b0bbbacde46","Type":"ContainerStarted","Data":"12c4e47da560a5a9e83465eb1ca6ebe0ac125df27be10abf409b9d372f4b517a"} Sep 18 22:02:27.467367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:27.467315 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v98jj" podStartSLOduration=130.657202628 podStartE2EDuration="2m12.46730054s" totalImagesPullingTime="1.810097912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:27.466105907 +0000 UTC m=+133.137603894" watchObservedRunningTime="2026-09-18 22:02:27.46730054 +0000 UTC m=+133.138798532" Sep 18 22:02:27.601154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:27.601129 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:27.694555 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:27.694466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:28.876880 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:28.876814 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b4f8688c8-czv2p" podUID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" containerName="console" containerID="cri-o://cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d" gracePeriod=15 Sep 18 22:02:29.159960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.159934 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:02:29.191945 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.191917 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh7qt\" (UniqueName: \"kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.191951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.191989 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192096 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:02:29.192523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192460 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config\") pod \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\" (UID: \"9e2df326-6bef-45d0-a615-fc803b0dc1b9\") " Sep 18 22:02:29.192523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192468 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:02:29.192523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192473 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config" (OuterVolumeSpecName: "console-config") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:02:29.192806 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192744 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-config\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.192806 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192771 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.193045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.192806 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e2df326-6bef-45d0-a615-fc803b0dc1b9-oauth-serving-cert\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.194558 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.194534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:02:29.194656 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.194573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt" (OuterVolumeSpecName: "kube-api-access-zh7qt") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "kube-api-access-zh7qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:02:29.194939 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.194921 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9e2df326-6bef-45d0-a615-fc803b0dc1b9" (UID: "9e2df326-6bef-45d0-a615-fc803b0dc1b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:02:29.294052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.294016 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-serving-cert\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.294052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.294052 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e2df326-6bef-45d0-a615-fc803b0dc1b9-console-oauth-config\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.294233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.294066 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7qt\" (UniqueName: \"kubernetes.io/projected/9e2df326-6bef-45d0-a615-fc803b0dc1b9-kube-api-access-zh7qt\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:29.442004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.441974 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" containerID="cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d" exitCode=0 Sep 18 22:02:29.442137 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.442010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4f8688c8-czv2p" event={"ID":"9e2df326-6bef-45d0-a615-fc803b0dc1b9","Type":"ContainerDied","Data":"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d"} Sep 18 22:02:29.442137 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.442019 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4f8688c8-czv2p" Sep 18 22:02:29.442137 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.442048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4f8688c8-czv2p" event={"ID":"9e2df326-6bef-45d0-a615-fc803b0dc1b9","Type":"ContainerDied","Data":"e65ee6029c5f271527b81526acc58d5e7fd5125d5b53fe3273577a2b0c8a5aa1"} Sep 18 22:02:29.442137 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.442067 2577 scope.go:117] "RemoveContainer" containerID="cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d" Sep 18 22:02:29.451189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.451171 2577 scope.go:117] "RemoveContainer" containerID="cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d" Sep 18 22:02:29.451472 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:02:29.451453 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d\": container with ID starting with cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d not found: ID does not exist" containerID="cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d" Sep 18 22:02:29.451519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.451481 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d"} err="failed to get container status \"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d\": rpc error: code = NotFound desc = could not find container \"cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d\": container with ID starting with cf8f120ecc253afa8dcf3e14dda170880f1114032574e14838cd044da317763d not found: ID does not exist" Sep 18 22:02:29.481138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.481111 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:02:29.485616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:29.485596 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b4f8688c8-czv2p"] Sep 18 22:02:30.846514 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:30.846471 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" path="/var/lib/kubelet/pods/9e2df326-6bef-45d0-a615-fc803b0dc1b9/volumes" Sep 18 22:02:31.889504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:31.889467 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" podUID="a7badbe7-0f56-4946-9961-f9e3900a560f" containerName="registry" containerID="cri-o://a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7" gracePeriod=30 Sep 18 22:02:32.180696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.180670 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:02:32.259997 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.259972 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:32.260144 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.260051 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:32.264619 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.264602 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:32.317011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.316992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkbqv\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317024 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317116 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317130 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317186 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets\") pod \"a7badbe7-0f56-4946-9961-f9e3900a560f\" (UID: \"a7badbe7-0f56-4946-9961-f9e3900a560f\") " Sep 18 22:02:32.317821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.317617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:02:32.318435 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.318408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:02:32.320211 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.319940 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:02:32.320211 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.320016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:02:32.320398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.320376 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv" (OuterVolumeSpecName: "kube-api-access-rkbqv") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "kube-api-access-rkbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:02:32.320464 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.320445 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:02:32.320657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.320637 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:02:32.325721 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.325698 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a7badbe7-0f56-4946-9961-f9e3900a560f" (UID: "a7badbe7-0f56-4946-9961-f9e3900a560f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:02:32.418076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418029 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbqv\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-kube-api-access-rkbqv\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418050 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-image-registry-private-configuration\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418060 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-trusted-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418069 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7badbe7-0f56-4946-9961-f9e3900a560f-ca-trust-extracted\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418078 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418086 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7badbe7-0f56-4946-9961-f9e3900a560f-registry-certificates\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418094 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7badbe7-0f56-4946-9961-f9e3900a560f-bound-sa-token\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.418257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.418101 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7badbe7-0f56-4946-9961-f9e3900a560f-installation-pull-secrets\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:02:32.455948 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.455924 2577 generic.go:356] "Generic (PLEG): container finished" podID="a7badbe7-0f56-4946-9961-f9e3900a560f" containerID="a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7" exitCode=0 Sep 18 22:02:32.456057 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.455976 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" Sep 18 22:02:32.456057 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.456028 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" event={"ID":"a7badbe7-0f56-4946-9961-f9e3900a560f","Type":"ContainerDied","Data":"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7"} Sep 18 22:02:32.456216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.456069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5b64f97c-rvj8p" event={"ID":"a7badbe7-0f56-4946-9961-f9e3900a560f","Type":"ContainerDied","Data":"e68e19c4362ea32a6ad7819ac5cb65197cf37554064667d53e4b8c8ce8e8ffcc"} Sep 18 22:02:32.456216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.456092 2577 scope.go:117] "RemoveContainer" containerID="a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7" Sep 18 22:02:32.460376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.460356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:02:32.464520 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.464507 2577 scope.go:117] "RemoveContainer" containerID="a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7" Sep 18 22:02:32.464774 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:02:32.464757 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7\": container with ID starting with a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7 not found: ID does not exist" containerID="a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7" Sep 18 22:02:32.464893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.464780 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7"} err="failed to get container status \"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7\": rpc error: code = NotFound desc = could not find container \"a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7\": container with ID starting with a11d4d265340e23f396ee8e92be78444e705b798efea1a0517dc83bee9d47bb7 not found: ID does not exist" Sep 18 22:02:32.510875 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.510851 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:02:32.521378 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.521356 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c5b64f97c-rvj8p"] Sep 18 22:02:32.719446 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.719412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:32.847401 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.847372 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7badbe7-0f56-4946-9961-f9e3900a560f" path="/var/lib/kubelet/pods/a7badbe7-0f56-4946-9961-f9e3900a560f/volumes" Sep 18 22:02:32.853703 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:32.853672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:33.010074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:33.010020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:33.184763 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:33.184718 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:34.382271 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:34.382241 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:34.382622 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:34.382297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:36.127993 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:36.127958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:36.224061 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:36.224008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:42.097643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:42.097607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:42.255008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:42.254979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:02:54.386679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:54.386612 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:02:54.390561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:02:54.390536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-774dc96fdf-k8sdr" Sep 18 22:03:16.480337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:16.480303 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:03:16.499836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:16.499813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:03:16.647073 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:16.647036 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 22:03:26.116030 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:26.115999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:26.205581 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:26.205549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:35.373019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:35.372985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:35.463555 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:35.463529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:38.058404 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:38.058369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:38.158769 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:38.158737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:38.673320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:38.673273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:42.028291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:42.028234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:42.129540 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:42.129511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:46.280396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:46.280355 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:03:46.414928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:46.414894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:46.519088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:46.519057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:46.615538 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:46.615453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:55.890747 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:55.890716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:59.719573 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:59.719541 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:03:59.799021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:03:59.798998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:04:00.968479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:00.968438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:04:01.061004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:01.060966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:04:11.299685 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.299615 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bd4559cfc-cj5vk" podUID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" containerName="console" containerID="cri-o://3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f" gracePeriod=15 Sep 18 22:04:11.545670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.545647 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:04:11.642437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642366 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642556 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642556 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642505 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642556 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642530 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642718 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642560 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642718 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642600 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642718 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44f8q\" (UniqueName: \"kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q\") pod \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\" (UID: \"ae73299b-0550-4e91-b977-2afc9f5bc7f2\") " Sep 18 22:04:11.642903 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642748 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca" (OuterVolumeSpecName: "service-ca") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:04:11.642960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642896 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:04:11.642960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.642943 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:04:11.643043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.643013 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-oauth-serving-cert\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.643043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.643032 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.643298 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.643274 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config" (OuterVolumeSpecName: "console-config") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:04:11.644760 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.644714 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:04:11.644888 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.644842 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:04:11.645142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.645117 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q" (OuterVolumeSpecName: "kube-api-access-44f8q") pod "ae73299b-0550-4e91-b977-2afc9f5bc7f2" (UID: "ae73299b-0550-4e91-b977-2afc9f5bc7f2"). InnerVolumeSpecName "kube-api-access-44f8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:04:11.744280 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.744256 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-config\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.744280 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.744277 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-serving-cert\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.744442 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.744287 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f8q\" (UniqueName: \"kubernetes.io/projected/ae73299b-0550-4e91-b977-2afc9f5bc7f2-kube-api-access-44f8q\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.744442 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.744296 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73299b-0550-4e91-b977-2afc9f5bc7f2-trusted-ca-bundle\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.744442 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.744304 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae73299b-0550-4e91-b977-2afc9f5bc7f2-console-oauth-config\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:04:11.839530 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.839505 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" containerID="3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f" exitCode=0 Sep 18 22:04:11.839646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.839545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd4559cfc-cj5vk" event={"ID":"ae73299b-0550-4e91-b977-2afc9f5bc7f2","Type":"ContainerDied","Data":"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f"} Sep 18 22:04:11.839646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.839562 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd4559cfc-cj5vk" Sep 18 22:04:11.839646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.839593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd4559cfc-cj5vk" event={"ID":"ae73299b-0550-4e91-b977-2afc9f5bc7f2","Type":"ContainerDied","Data":"f3dc3a316d1d1d45e49b7c5dccc9d4ae805d56366935755f5da51a2773e3f893"} Sep 18 22:04:11.839646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.839612 2577 scope.go:117] "RemoveContainer" containerID="3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f" Sep 18 22:04:11.848123 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.848109 2577 scope.go:117] "RemoveContainer" containerID="3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f" Sep 18 22:04:11.848391 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:04:11.848370 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f\": container with ID starting with 3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f not found: ID does not exist" containerID="3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f" Sep 18 22:04:11.848436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.848397 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f"} err="failed to get container status \"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f\": rpc error: code = NotFound desc = could not find container \"3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f\": container with ID starting with 3c8259f9dcd1ccae8c00c7d5ad2319f5c6f45260c4fd28a807d692a9118b888f not found: ID does not exist" Sep 18 22:04:11.860490 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.860466 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:04:11.864403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:11.864379 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bd4559cfc-cj5vk"] Sep 18 22:04:12.847052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:12.847024 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" path="/var/lib/kubelet/pods/ae73299b-0550-4e91-b977-2afc9f5bc7f2/volumes" Sep 18 22:04:51.147316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:04:51.147211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:05:14.751000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:05:14.750967 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 22:05:16.453905 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:05:16.453868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:05:55.378001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:05:55.377952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:05:56.631136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:05:56.631080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:02.702172 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:02.702138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:03.556219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:03.556177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:07.038177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:07.038066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:07.261547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:07.261511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:08.593400 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.593369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:08.731078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.731047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:08.784321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784294 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n2nm6"] Sep 18 22:06:08.784586 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784574 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7badbe7-0f56-4946-9961-f9e3900a560f" containerName="registry" Sep 18 22:06:08.784629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784588 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7badbe7-0f56-4946-9961-f9e3900a560f" containerName="registry" Sep 18 22:06:08.784629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784599 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" containerName="console" Sep 18 22:06:08.784629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784605 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" containerName="console" Sep 18 22:06:08.784629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784613 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" containerName="console" Sep 18 22:06:08.784629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784618 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" containerName="console" Sep 18 22:06:08.784772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784672 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73299b-0550-4e91-b977-2afc9f5bc7f2" containerName="console" Sep 18 22:06:08.784772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784682 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7badbe7-0f56-4946-9961-f9e3900a560f" containerName="registry" Sep 18 22:06:08.784772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.784689 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2df326-6bef-45d0-a615-fc803b0dc1b9" containerName="console" Sep 18 22:06:08.787444 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.787431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:08.790075 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.790056 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 18 22:06:08.790197 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.790144 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 18 22:06:08.808663 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.808634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n2nm6"] Sep 18 22:06:08.884267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.884199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbq4n\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-kube-api-access-gbq4n\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:08.884267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.884258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:08.985250 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.985226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:08.985366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:08.985278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbq4n\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-kube-api-access-gbq4n\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:09.003363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.003344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:09.003457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.003382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbq4n\" (UniqueName: \"kubernetes.io/projected/6c86a868-48f9-463b-813f-922ef9f18974-kube-api-access-gbq4n\") pod \"cert-manager-webhook-6486ffd997-n2nm6\" (UID: \"6c86a868-48f9-463b-813f-922ef9f18974\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:09.086280 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.086246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:09.104204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.104182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:09.207308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.207280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:09.390429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.390400 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n2nm6"] Sep 18 22:06:09.394183 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:06:09.394160 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c86a868_48f9_463b_813f_922ef9f18974.slice/crio-0e9b5f9bf1d2f7001acf5636031e6d2dae5e4b4de2985250e7c1711f3ce18f32 WatchSource:0}: Error finding container 0e9b5f9bf1d2f7001acf5636031e6d2dae5e4b4de2985250e7c1711f3ce18f32: Status 404 returned error can't find the container with id 0e9b5f9bf1d2f7001acf5636031e6d2dae5e4b4de2985250e7c1711f3ce18f32 Sep 18 22:06:09.396062 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:09.396045 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:06:10.293440 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:10.293403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" event={"ID":"6c86a868-48f9-463b-813f-922ef9f18974","Type":"ContainerStarted","Data":"0e9b5f9bf1d2f7001acf5636031e6d2dae5e4b4de2985250e7c1711f3ce18f32"} Sep 18 22:06:10.950732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:10.950700 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:11.282014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:11.281907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:13.308567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:13.308533 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" event={"ID":"6c86a868-48f9-463b-813f-922ef9f18974","Type":"ContainerStarted","Data":"df836033fdf48c39aab95876d8949ec2af0f9152d3432f97414dbccdcb698ee0"} Sep 18 22:06:13.308951 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:13.308581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:13.330159 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:13.330117 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" podStartSLOduration=2.06349436 podStartE2EDuration="5.330103187s" totalImagesPullingTime="3.266608827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:06:13.328985649 +0000 UTC m=+359.000483618" watchObservedRunningTime="2026-09-18 22:06:13.330103187 +0000 UTC m=+359.001601156" Sep 18 22:06:18.191296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:18.191267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:19.316416 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:19.316380 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n2nm6" Sep 18 22:06:19.818906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:19.818876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:19.917766 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:19.917737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:25.482642 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:25.482608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:25.585182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:25.585153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:27.955754 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:27.955730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:38.968064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:38.968037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:39.083783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:39.083754 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:39.629889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:39.629863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.049917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.049888 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.162277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.162251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.229014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.228981 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch"] Sep 18 22:06:40.232236 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.232218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.234605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.234585 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lkk4s\"" Sep 18 22:06:40.234725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.234588 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 22:06:40.234808 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.234759 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 22:06:40.234879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.234824 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 22:06:40.234879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.234868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 22:06:40.251616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.251594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch"] Sep 18 22:06:40.410632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.410550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.414767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.414727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-tmp\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.415013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.414991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hgf\" (UniqueName: \"kubernetes.io/projected/49479021-14f9-46e4-8185-9fe81e7fe5e0-kube-api-access-l8hgf\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.415141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.415126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-manifests\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.415261 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.415248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.415373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.415360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.515739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.515711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-manifests\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.515917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.515758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.515917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.515783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.515917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.515842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-tmp\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.515917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.515884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hgf\" (UniqueName: \"kubernetes.io/projected/49479021-14f9-46e4-8185-9fe81e7fe5e0-kube-api-access-l8hgf\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.516106 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.516015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.516245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.516212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-manifests\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.518274 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.518247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49479021-14f9-46e4-8185-9fe81e7fe5e0-tmp\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.518357 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.518325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.518503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.518487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49479021-14f9-46e4-8185-9fe81e7fe5e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.525737 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.525717 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hgf\" (UniqueName: \"kubernetes.io/projected/49479021-14f9-46e4-8185-9fe81e7fe5e0-kube-api-access-l8hgf\") pod \"opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch\" (UID: \"49479021-14f9-46e4-8185-9fe81e7fe5e0\") " pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.542544 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.542522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:40.688708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.688679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.835562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.835526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:40.849436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.849405 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch"] Sep 18 22:06:40.852584 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:06:40.852557 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49479021_14f9_46e4_8185_9fe81e7fe5e0.slice/crio-3a083f2ae55f65481c487496a713c1f8714aa9e91eb81a2be4acff28528fa8d6 WatchSource:0}: Error finding container 3a083f2ae55f65481c487496a713c1f8714aa9e91eb81a2be4acff28528fa8d6: Status 404 returned error can't find the container with id 3a083f2ae55f65481c487496a713c1f8714aa9e91eb81a2be4acff28528fa8d6 Sep 18 22:06:40.926990 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:40.926962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:41.046637 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:41.046555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:41.422218 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:41.422130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" event={"ID":"49479021-14f9-46e4-8185-9fe81e7fe5e0","Type":"ContainerStarted","Data":"3a083f2ae55f65481c487496a713c1f8714aa9e91eb81a2be4acff28528fa8d6"} Sep 18 22:06:44.437016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:44.436981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" event={"ID":"49479021-14f9-46e4-8185-9fe81e7fe5e0","Type":"ContainerStarted","Data":"6a9f4b74fb6f87e901bdf6d123dcdcad709d1bb04d84e199f46127f011d721fc"} Sep 18 22:06:44.935646 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:44.935620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:45.048367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:45.048338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:45.443532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:45.443498 2577 generic.go:356] "Generic (PLEG): container finished" podID="49479021-14f9-46e4-8185-9fe81e7fe5e0" containerID="6a9f4b74fb6f87e901bdf6d123dcdcad709d1bb04d84e199f46127f011d721fc" exitCode=0 Sep 18 22:06:45.443980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:45.443576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" event={"ID":"49479021-14f9-46e4-8185-9fe81e7fe5e0","Type":"ContainerDied","Data":"6a9f4b74fb6f87e901bdf6d123dcdcad709d1bb04d84e199f46127f011d721fc"} Sep 18 22:06:46.449561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:46.449526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" event={"ID":"49479021-14f9-46e4-8185-9fe81e7fe5e0","Type":"ContainerStarted","Data":"f0a8d1556b67ad21e8a2a4b1d30c1997ae21d9d58481247d7b2eefa4f53f5ed7"} Sep 18 22:06:46.449951 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:46.449675 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:46.476746 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:46.476688 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" podStartSLOduration=1.877686151 podStartE2EDuration="6.476657115s" totalImagesPullingTime="3.581299755s" totalInitContainerRuntime="1.017671209s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:06:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:06:46.474524377 +0000 UTC m=+392.146022358" watchObservedRunningTime="2026-09-18 22:06:46.476657115 +0000 UTC m=+392.148155086" Sep 18 22:06:54.246174 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:54.246134 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:54.360225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:54.360148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:55.301384 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:55.301337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:55.365262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:55.365240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:57.456660 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:57.456631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch" Sep 18 22:06:59.422876 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:59.422848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:59.568461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:59.568431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:06:59.932455 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:06:59.932425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:00.061843 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.061806 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh"] Sep 18 22:07:00.069991 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.069964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.072485 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.072467 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 22:07:00.073245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.073227 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 18 22:07:00.073322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.073240 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 18 22:07:00.073322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.073313 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l7zbn\"" Sep 18 22:07:00.073439 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.073341 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 22:07:00.079848 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.079827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh"] Sep 18 22:07:00.080749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.080732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:00.157864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.157838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55jw\" (UniqueName: \"kubernetes.io/projected/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-kube-api-access-h55jw\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.158022 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.157870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tls-certs\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.158022 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.157887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tmp\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.259164 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.259090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55jw\" (UniqueName: \"kubernetes.io/projected/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-kube-api-access-h55jw\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.259164 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.259127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tls-certs\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.259164 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.259149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tmp\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.261447 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.261426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tmp\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.261769 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.261747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-tls-certs\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.272304 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.272284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55jw\" (UniqueName: \"kubernetes.io/projected/c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9-kube-api-access-h55jw\") pod \"kube-auth-proxy-666584dbdf-v2qxh\" (UID: \"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9\") " pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.380121 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.380095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" Sep 18 22:07:00.561416 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.561332 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl"] Sep 18 22:07:00.565083 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.565061 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.567894 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.567876 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 22:07:00.568097 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.568064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7t69b\"" Sep 18 22:07:00.568169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.568099 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 22:07:00.568969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.568953 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 22:07:00.569039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.568956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 18 22:07:00.569039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.568997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 22:07:00.579525 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.579507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl"] Sep 18 22:07:00.661734 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.661706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.661904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.661744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-metrics-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.661904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.661820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fda16389-960b-4497-93da-6a882590cbee-manager-config\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.661904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.661856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnks\" (UniqueName: \"kubernetes.io/projected/fda16389-960b-4497-93da-6a882590cbee-kube-api-access-zqnks\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.672085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.672061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh"] Sep 18 22:07:00.675342 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:07:00.675317 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cb7e18_5fa7_4ed6_aa4e_70fe8b9025a9.slice/crio-9857fadaf81ab62fc4c1b54372402e7e7278b4c375ed4576a6e397a4e5c2c72b WatchSource:0}: Error finding container 9857fadaf81ab62fc4c1b54372402e7e7278b4c375ed4576a6e397a4e5c2c72b: Status 404 returned error can't find the container with id 9857fadaf81ab62fc4c1b54372402e7e7278b4c375ed4576a6e397a4e5c2c72b Sep 18 22:07:00.762649 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.762623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.762759 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.762657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-metrics-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.762759 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.762689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fda16389-960b-4497-93da-6a882590cbee-manager-config\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.762759 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.762718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnks\" (UniqueName: \"kubernetes.io/projected/fda16389-960b-4497-93da-6a882590cbee-kube-api-access-zqnks\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.763432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.763366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fda16389-960b-4497-93da-6a882590cbee-manager-config\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.765254 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.765234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.765329 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.765262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda16389-960b-4497-93da-6a882590cbee-metrics-cert\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.780605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.780584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnks\" (UniqueName: \"kubernetes.io/projected/fda16389-960b-4497-93da-6a882590cbee-kube-api-access-zqnks\") pod \"lws-controller-manager-5fd7755fbb-nfqcl\" (UID: \"fda16389-960b-4497-93da-6a882590cbee\") " pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:00.874323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:00.874276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:01.171569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:01.171545 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl"] Sep 18 22:07:01.172452 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:07:01.172423 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda16389_960b_4497_93da_6a882590cbee.slice/crio-b8894bbb3d020a0ebc1d2f9d0411f2138d6de62e8c4a9d3c3e9d43c78da2f581 WatchSource:0}: Error finding container b8894bbb3d020a0ebc1d2f9d0411f2138d6de62e8c4a9d3c3e9d43c78da2f581: Status 404 returned error can't find the container with id b8894bbb3d020a0ebc1d2f9d0411f2138d6de62e8c4a9d3c3e9d43c78da2f581 Sep 18 22:07:01.517492 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:01.517454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" event={"ID":"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9","Type":"ContainerStarted","Data":"9857fadaf81ab62fc4c1b54372402e7e7278b4c375ed4576a6e397a4e5c2c72b"} Sep 18 22:07:01.518876 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:01.518847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" event={"ID":"fda16389-960b-4497-93da-6a882590cbee","Type":"ContainerStarted","Data":"b8894bbb3d020a0ebc1d2f9d0411f2138d6de62e8c4a9d3c3e9d43c78da2f581"} Sep 18 22:07:01.857060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:01.856977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:01.931653 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:01.931619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:03.095246 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:03.095201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:05.541519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:05.541483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" event={"ID":"c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9","Type":"ContainerStarted","Data":"f6efd672b575e722f3325baead3f4e0deccc1bdc809472f056a8b424106c627c"} Sep 18 22:07:05.543088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:05.543065 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" event={"ID":"fda16389-960b-4497-93da-6a882590cbee","Type":"ContainerStarted","Data":"4d342c8d12e8e532c9dc0d0f6d499f2056d2ea28e80768e19a7f1abc8522ba8c"} Sep 18 22:07:05.543182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:05.543133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:05.563210 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:05.563166 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-666584dbdf-v2qxh" podStartSLOduration=1.580970678 podStartE2EDuration="5.563151606s" totalImagesPullingTime="3.982180928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:07:05.56221272 +0000 UTC m=+411.233710691" watchObservedRunningTime="2026-09-18 22:07:05.563151606 +0000 UTC m=+411.234649576" Sep 18 22:07:05.585588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:05.585546 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" podStartSLOduration=2.052050378 podStartE2EDuration="5.585533535s" totalImagesPullingTime="3.533483157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:07:05.584185274 +0000 UTC m=+411.255683225" watchObservedRunningTime="2026-09-18 22:07:05.585533535 +0000 UTC m=+411.257031506" Sep 18 22:07:06.066676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:06.066646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:06.167227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:06.167200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:06.253475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:06.253450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:09.987076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:09.987042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:10.092186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:10.092159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:10.688843 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:10.688812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:10.795913 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:10.795876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:10.898915 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:10.898882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:11.054532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:11.054468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:11.158529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:11.158500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:11.261500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:11.261468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:11.362366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:11.362297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:12.083443 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:12.083412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:16.550333 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:16.550302 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5fd7755fbb-nfqcl" Sep 18 22:07:22.027256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:22.027228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:22.336432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:22.336333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:22.517529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:22.517496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:23.643576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:23.643542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:23.798608 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:23.798580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:07:23.990814 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:07:23.990737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:11.805715 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:11.805683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:12.017475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:12.017443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:30.057195 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:30.057163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:32.693387 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:32.693355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:33.076000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:33.075918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:33.540482 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:33.540451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:34.503869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:34.503836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:36.383423 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:36.383390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:36.591054 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:36.591017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:37.338390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:37.338364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:38.883979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:38.883949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:39.182073 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:39.182047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:39.284034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:39.284006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:41.562123 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:41.562094 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:43.128196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:43.128165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:43.261436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:43.261409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:45.386904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:45.386862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:45.715685 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:45.715658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:47.925263 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:47.925236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:48.033355 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:48.033325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:48.200271 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:48.200230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:48.445424 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:48.445393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:54.920092 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:54.920008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:55.042632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:55.042604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:59.719772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:59.719736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:59.847589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:59.847560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:08:59.940116 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:08:59.940071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:02.417065 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:02.417015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:02.519816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:02.519730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:07.692296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:07.692199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:07.794705 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:07.794655 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:08.098388 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:08.098320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:08.224011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:08.223986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:08.523986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:08.523961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:08.616253 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:08.616229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:14.700679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:14.700647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:14.802711 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:14.802679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:19.548473 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:19.548440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:19.728035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:19.728002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:28.145321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:28.145293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:28.411633 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:28.411551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:30.758475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.758442 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:30.768542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.768521 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:30.771972 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.771958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.775623 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.775603 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jxbsr\"" Sep 18 22:09:30.775739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.775650 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 18 22:09:30.776397 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.776379 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 18 22:09:30.776491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.776441 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 18 22:09:30.783653 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.783630 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:30.802357 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.802330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8phx\" (UniqueName: \"kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.802450 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.802399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.862126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.862094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:30.873184 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.873161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:30.903561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.903538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.903662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.903587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8phx\" (UniqueName: \"kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.904108 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.904091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.911971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.911952 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8phx\" (UniqueName: \"kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx\") pod \"limitador-limitador-8588749977-8s9q9\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:30.936299 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:30.936274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:31.082024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.081974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:31.147691 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.147630 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:09:31.151309 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.151288 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.162371 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.162334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:09:31.179550 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.179531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:09:31.205314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.205287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkf5\" (UniqueName: \"kubernetes.io/projected/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-kube-api-access-xzkf5\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.205394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.205322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-config-file\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.306572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.306548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkf5\" (UniqueName: \"kubernetes.io/projected/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-kube-api-access-xzkf5\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.306652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.306582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-config-file\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.307181 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.307166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-config-file\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.316006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.315982 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkf5\" (UniqueName: \"kubernetes.io/projected/b14ab5ad-fd10-4ce2-870d-0cc26b06baf5-kube-api-access-xzkf5\") pod \"limitador-limitador-84994f7ffc-4kg8t\" (UID: \"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.355819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.355771 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:31.358318 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:09:31.358298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b4b997_43a9_4fe8_a070_2e17a644fcd0.slice/crio-25927ee0d0e5e38bcba993537035a7cc4eb6b7caa6715dc992e3c14f6d62c72a WatchSource:0}: Error finding container 25927ee0d0e5e38bcba993537035a7cc4eb6b7caa6715dc992e3c14f6d62c72a: Status 404 returned error can't find the container with id 25927ee0d0e5e38bcba993537035a7cc4eb6b7caa6715dc992e3c14f6d62c72a Sep 18 22:09:31.463356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.463338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:31.640019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.639932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:31.719562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.719520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:31.743668 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:31.743648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:09:31.746046 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:09:31.746019 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14ab5ad_fd10_4ce2_870d_0cc26b06baf5.slice/crio-6d61c332f02766c3c90dc2b2831334f5fac0af42a01fb6b8a87babd0b832a265 WatchSource:0}: Error finding container 6d61c332f02766c3c90dc2b2831334f5fac0af42a01fb6b8a87babd0b832a265: Status 404 returned error can't find the container with id 6d61c332f02766c3c90dc2b2831334f5fac0af42a01fb6b8a87babd0b832a265 Sep 18 22:09:32.155364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:32.155329 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" event={"ID":"c0b4b997-43a9-4fe8-a070-2e17a644fcd0","Type":"ContainerStarted","Data":"25927ee0d0e5e38bcba993537035a7cc4eb6b7caa6715dc992e3c14f6d62c72a"} Sep 18 22:09:32.156809 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:32.156746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" event={"ID":"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5","Type":"ContainerStarted","Data":"6d61c332f02766c3c90dc2b2831334f5fac0af42a01fb6b8a87babd0b832a265"} Sep 18 22:09:34.169302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.169206 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" event={"ID":"c0b4b997-43a9-4fe8-a070-2e17a644fcd0","Type":"ContainerStarted","Data":"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e"} Sep 18 22:09:34.169701 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.169376 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:34.170781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.170764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" event={"ID":"b14ab5ad-fd10-4ce2-870d-0cc26b06baf5","Type":"ContainerStarted","Data":"5abd29205689c90ea9c045bab89bd11795a162ef67e9fe62ec523e112bed5b8b"} Sep 18 22:09:34.170911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.170899 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:34.237819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.237766 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" podStartSLOduration=1.127826327 podStartE2EDuration="3.237753588s" totalImagesPullingTime="2.109927261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:09:34.236431203 +0000 UTC m=+559.907929192" watchObservedRunningTime="2026-09-18 22:09:34.237753588 +0000 UTC m=+559.909251558" Sep 18 22:09:34.238627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:34.238592 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" podStartSLOduration=1.746505973 podStartE2EDuration="4.238580237s" totalImagesPullingTime="2.492074264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:09:34.208129408 +0000 UTC m=+559.879627400" watchObservedRunningTime="2026-09-18 22:09:34.238580237 +0000 UTC m=+559.910078208" Sep 18 22:09:35.147066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:35.147025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:35.247663 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:35.247604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:45.176111 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.176084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4kg8t" Sep 18 22:09:45.176479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.176146 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:45.271141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.271113 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:45.271294 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.271269 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" podUID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" containerName="limitador" containerID="cri-o://128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e" gracePeriod=30 Sep 18 22:09:45.813386 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.813363 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:45.920013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.919942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file\") pod \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " Sep 18 22:09:45.920217 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.920029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8phx\" (UniqueName: \"kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx\") pod \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\" (UID: \"c0b4b997-43a9-4fe8-a070-2e17a644fcd0\") " Sep 18 22:09:45.920336 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.920311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file" (OuterVolumeSpecName: "config-file") pod "c0b4b997-43a9-4fe8-a070-2e17a644fcd0" (UID: "c0b4b997-43a9-4fe8-a070-2e17a644fcd0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:09:45.922254 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:45.922225 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx" (OuterVolumeSpecName: "kube-api-access-p8phx") pod "c0b4b997-43a9-4fe8-a070-2e17a644fcd0" (UID: "c0b4b997-43a9-4fe8-a070-2e17a644fcd0"). InnerVolumeSpecName "kube-api-access-p8phx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:09:46.020662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.020643 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8phx\" (UniqueName: \"kubernetes.io/projected/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-kube-api-access-p8phx\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:09:46.020774 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.020666 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b4b997-43a9-4fe8-a070-2e17a644fcd0-config-file\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:09:46.229092 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.229066 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" containerID="128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e" exitCode=0 Sep 18 22:09:46.229468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.229105 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" Sep 18 22:09:46.229468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.229158 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" event={"ID":"c0b4b997-43a9-4fe8-a070-2e17a644fcd0","Type":"ContainerDied","Data":"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e"} Sep 18 22:09:46.229468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.229204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8s9q9" event={"ID":"c0b4b997-43a9-4fe8-a070-2e17a644fcd0","Type":"ContainerDied","Data":"25927ee0d0e5e38bcba993537035a7cc4eb6b7caa6715dc992e3c14f6d62c72a"} Sep 18 22:09:46.229468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.229224 2577 scope.go:117] "RemoveContainer" containerID="128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e" Sep 18 22:09:46.238083 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.238068 2577 scope.go:117] "RemoveContainer" containerID="128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e" Sep 18 22:09:46.238402 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:09:46.238385 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e\": container with ID starting with 128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e not found: ID does not exist" containerID="128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e" Sep 18 22:09:46.238456 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.238410 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e"} err="failed to get container status \"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e\": rpc error: code = NotFound desc = could not find container \"128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e\": container with ID starting with 128493e039dd5e3d08a4be229866e25fe88d7f864c6ec083244f9a6b67a2289e not found: ID does not exist" Sep 18 22:09:46.252499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.252479 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:46.256720 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.256702 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8s9q9"] Sep 18 22:09:46.848046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.848008 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" path="/var/lib/kubelet/pods/c0b4b997-43a9-4fe8-a070-2e17a644fcd0/volumes" Sep 18 22:09:46.871655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.871624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:46.970821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:46.970783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:47.407272 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:47.407244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:59.254388 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:59.254357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:09:59.710019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:09:59.709994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:06.034810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.034761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:06.171064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.171038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:06.279132 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.279108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:06.346454 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.346388 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:10:06.346717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.346706 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" containerName="limitador" Sep 18 22:10:06.346758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.346719 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" containerName="limitador" Sep 18 22:10:06.346805 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.346767 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b4b997-43a9-4fe8-a070-2e17a644fcd0" containerName="limitador" Sep 18 22:10:06.354044 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.354026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.361477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.361451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 22:10:06.361575 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.361451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 22:10:06.361575 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.361509 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 22:10:06.361685 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.361453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ww7bb\"" Sep 18 22:10:06.376356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.376333 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:10:06.398976 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.398957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:06.468774 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.468750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.468883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.468812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtz8g\" (UniqueName: \"kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.468883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.468865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.569499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.569479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.569621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.569524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.569621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.569555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtz8g\" (UniqueName: \"kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.571866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.571847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.571962 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.571877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.587317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.587295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtz8g\" (UniqueName: \"kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g\") pod \"maas-controller-7b57d79649-fdn2s\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.663896 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.663848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:06.942344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:06.942323 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:10:06.944862 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:10:06.944835 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68522970_3c26_4ead_aa07_0b3f14f9577f.slice/crio-1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8 WatchSource:0}: Error finding container 1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8: Status 404 returned error can't find the container with id 1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8 Sep 18 22:10:07.329269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:07.329194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d79649-fdn2s" event={"ID":"68522970-3c26-4ead-aa07-0b3f14f9577f","Type":"ContainerStarted","Data":"1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8"} Sep 18 22:10:09.268219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:09.268185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:09.340269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:09.340235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d79649-fdn2s" event={"ID":"68522970-3c26-4ead-aa07-0b3f14f9577f","Type":"ContainerStarted","Data":"6f8022de28d6f016859c0d0fbd295afde6e7d9a41b69d88a227a773676afdf34"} Sep 18 22:10:09.340390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:09.340313 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:09.371561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:09.371494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:09.387457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:09.387412 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b57d79649-fdn2s" podStartSLOduration=1.239090836 podStartE2EDuration="3.38739891s" totalImagesPullingTime="2.148308074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:10:09.385198677 +0000 UTC m=+595.056696650" watchObservedRunningTime="2026-09-18 22:10:09.38739891 +0000 UTC m=+595.058896878" Sep 18 22:10:12.848081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:12.848023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:20.348220 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:20.348192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:10:40.964465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:40.964392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:41.239126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:41.239050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:42.537651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.537622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:42.822891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.822818 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:10:42.828695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.828666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.834648 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.834624 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 22:10:42.835342 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.835185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c6l9h\"" Sep 18 22:10:42.835683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.835667 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 22:10:42.835849 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.835704 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 22:10:42.841428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.841412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 22:10:42.847720 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.847699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.847844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.847731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.847844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.847753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.847954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.847886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xnj\" (UniqueName: \"kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.853208 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.853186 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:10:42.922959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.922939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:42.949479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.949453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.949579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.949505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.949579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.949544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.949695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.949637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xnj\" (UniqueName: \"kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.950698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.950669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.956152 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.956129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.957050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.957028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:42.977269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:42.977241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xnj\" (UniqueName: \"kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj\") pod \"maas-api-b89d7f5c7-tz79h\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:43.138816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:43.138729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:43.216698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:43.216666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:43.479883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:43.479856 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:10:43.482481 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:10:43.482459 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410b93f6_9c02_4051_8e7a_6279102c0183.slice/crio-b0b4a3bfa6a2084b7730822221703b69dbab6cf3f0b6215b0cb0c637149a699c WatchSource:0}: Error finding container b0b4a3bfa6a2084b7730822221703b69dbab6cf3f0b6215b0cb0c637149a699c: Status 404 returned error can't find the container with id b0b4a3bfa6a2084b7730822221703b69dbab6cf3f0b6215b0cb0c637149a699c Sep 18 22:10:43.493282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:43.493250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" event={"ID":"410b93f6-9c02-4051-8e7a-6279102c0183","Type":"ContainerStarted","Data":"b0b4a3bfa6a2084b7730822221703b69dbab6cf3f0b6215b0cb0c637149a699c"} Sep 18 22:10:44.062683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:44.062610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:44.288283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:44.288251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:44.614222 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:44.614182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:10:45.505655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:45.505623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" event={"ID":"410b93f6-9c02-4051-8e7a-6279102c0183","Type":"ContainerStarted","Data":"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11"} Sep 18 22:10:45.506073 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:45.505761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:45.537707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:45.537664 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" podStartSLOduration=2.020103329 podStartE2EDuration="3.537650954s" totalImagesPullingTime="1.517547625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:10:45.536262123 +0000 UTC m=+631.207760105" watchObservedRunningTime="2026-09-18 22:10:45.537650954 +0000 UTC m=+631.209148924" Sep 18 22:10:51.515153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:51.515125 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:10:52.679829 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:52.679780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:10:52.829662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:10:52.829632 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:08.137459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:08.137433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:09.238867 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:09.238827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:10.021477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:10.021425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:10.140475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:10.140441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:13.442324 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:13.442295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:13.540075 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:13.540049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:16.573918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.573883 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28"] Sep 18 22:11:16.577215 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.577198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.583369 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.583340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wjjvs\"" Sep 18 22:11:16.584294 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.584270 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 18 22:11:16.584391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.584278 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 22:11:16.584391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.584314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 22:11:16.624808 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.624768 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28"] Sep 18 22:11:16.660958 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.660930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:16.704148 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.704277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.704277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.704277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82829f42-260e-4cc1-badb-7df5b6e4b28a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.704393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9zc\" (UniqueName: \"kubernetes.io/projected/82829f42-260e-4cc1-badb-7df5b6e4b28a-kube-api-access-6x9zc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.704393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.704330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9zc\" (UniqueName: \"kubernetes.io/projected/82829f42-260e-4cc1-badb-7df5b6e4b28a-kube-api-access-6x9zc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805731 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805731 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805731 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.805918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82829f42-260e-4cc1-badb-7df5b6e4b28a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.806021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.805995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.806095 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.806074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.806154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.806127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.808231 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.808202 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82829f42-260e-4cc1-badb-7df5b6e4b28a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.808366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.808349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82829f42-260e-4cc1-badb-7df5b6e4b28a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.815696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.815677 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9zc\" (UniqueName: \"kubernetes.io/projected/82829f42-260e-4cc1-badb-7df5b6e4b28a-kube-api-access-6x9zc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28\" (UID: \"82829f42-260e-4cc1-badb-7df5b6e4b28a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:16.887678 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:16.887630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:17.164472 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:17.164392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:11:17.192052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:17.192024 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28"] Sep 18 22:11:17.194965 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:11:17.194936 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82829f42_260e_4cc1_badb_7df5b6e4b28a.slice/crio-82c522722b07b9f6812fcd5f12b6e899b2c10a49d978753647247cb12d40afd3 WatchSource:0}: Error finding container 82c522722b07b9f6812fcd5f12b6e899b2c10a49d978753647247cb12d40afd3: Status 404 returned error can't find the container with id 82c522722b07b9f6812fcd5f12b6e899b2c10a49d978753647247cb12d40afd3 Sep 18 22:11:17.196381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:17.196362 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:11:17.651295 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:17.651254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" event={"ID":"82829f42-260e-4cc1-badb-7df5b6e4b28a","Type":"ContainerStarted","Data":"82c522722b07b9f6812fcd5f12b6e899b2c10a49d978753647247cb12d40afd3"} Sep 18 22:11:18.657600 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:18.657521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" event={"ID":"82829f42-260e-4cc1-badb-7df5b6e4b28a","Type":"ContainerStarted","Data":"231b098f3abb11d41dbcb5515650e008827653ff996b4e6c3489e0465b0672f4"} Sep 18 22:11:18.692403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:18.692360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" podStartSLOduration=1.561715928 podStartE2EDuration="2.692321378s" totalImagesPullingTime="1.13060545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:11:18.691090096 +0000 UTC m=+664.362588099" watchObservedRunningTime="2026-09-18 22:11:18.692321378 +0000 UTC m=+664.363819347" Sep 18 22:11:26.887851 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:26.887821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:26.888284 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:26.887871 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:26.900489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:26.900461 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:27.710425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:27.710395 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28" Sep 18 22:11:32.345754 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:32.345716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:32.911208 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:32.911176 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:11:35.068112 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:35.068081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:56.624657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.624627 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6"] Sep 18 22:11:56.630849 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.630834 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.633346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.633326 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 18 22:11:56.638007 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.637986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6"] Sep 18 22:11:56.644036 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.644004 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:11:56.734688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94dx\" (UniqueName: \"kubernetes.io/projected/b3092fc1-9de2-4c44-bca8-45173c8fa184-kube-api-access-s94dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.734842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.734842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.734842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.734982 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3092fc1-9de2-4c44-bca8-45173c8fa184-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.734982 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.734908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3092fc1-9de2-4c44-bca8-45173c8fa184-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94dx\" (UniqueName: \"kubernetes.io/projected/b3092fc1-9de2-4c44-bca8-45173c8fa184-kube-api-access-s94dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.835986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.835944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.836230 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.836201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.836398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.836244 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.836491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.836291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.838237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.838215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3092fc1-9de2-4c44-bca8-45173c8fa184-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.838332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.838316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3092fc1-9de2-4c44-bca8-45173c8fa184-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.845484 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.845459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94dx\" (UniqueName: \"kubernetes.io/projected/b3092fc1-9de2-4c44-bca8-45173c8fa184-kube-api-access-s94dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6\" (UID: \"b3092fc1-9de2-4c44-bca8-45173c8fa184\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:56.942478 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:56.942448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:11:57.216292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:57.216270 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6"] Sep 18 22:11:57.219071 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:11:57.219046 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3092fc1_9de2_4c44_bca8_45173c8fa184.slice/crio-3f667caf812420dfa4a758a155470b98445700b5b611a5416f1a8bef6bc52bb1 WatchSource:0}: Error finding container 3f667caf812420dfa4a758a155470b98445700b5b611a5416f1a8bef6bc52bb1: Status 404 returned error can't find the container with id 3f667caf812420dfa4a758a155470b98445700b5b611a5416f1a8bef6bc52bb1 Sep 18 22:11:57.606745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:57.606665 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:11:57.841968 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:57.841930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" event={"ID":"b3092fc1-9de2-4c44-bca8-45173c8fa184","Type":"ContainerStarted","Data":"af7e6108ff7992ea75aa0ddc578de912d139c01da7792998b34a9fbc574062d5"} Sep 18 22:11:57.842317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:57.841975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" event={"ID":"b3092fc1-9de2-4c44-bca8-45173c8fa184","Type":"ContainerStarted","Data":"3f667caf812420dfa4a758a155470b98445700b5b611a5416f1a8bef6bc52bb1"} Sep 18 22:11:57.863847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:11:57.863724 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" podStartSLOduration=1.69433742 podStartE2EDuration="1.863710166s" totalImagesPullingTime="169.372746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:11:57.86185804 +0000 UTC m=+703.533356020" watchObservedRunningTime="2026-09-18 22:11:57.863710166 +0000 UTC m=+703.535208135" Sep 18 22:12:06.942903 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:06.942807 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:12:06.942903 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:06.942876 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:12:06.955272 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:06.955242 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:12:07.898297 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:07.898263 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6" Sep 18 22:12:08.289409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:08.289359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:12:09.420528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:09.420485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:12:12.865739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.865694 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb"] Sep 18 22:12:12.869362 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.869337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.872759 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.872734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 18 22:12:12.886740 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.886706 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb"] Sep 18 22:12:12.954426 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.954389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:12:12.977146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.977341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.977341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede078d3-2454-4e28-baf7-fb9a25829f3a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.977341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxdv\" (UniqueName: \"kubernetes.io/projected/ede078d3-2454-4e28-baf7-fb9a25829f3a-kube-api-access-vjxdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.977507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:12.977628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:12.977608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078653 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede078d3-2454-4e28-baf7-fb9a25829f3a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.078904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.078847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxdv\" (UniqueName: \"kubernetes.io/projected/ede078d3-2454-4e28-baf7-fb9a25829f3a-kube-api-access-vjxdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.079213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.079188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.079282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.079223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.079282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.079267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.081459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.081426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede078d3-2454-4e28-baf7-fb9a25829f3a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.081766 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.081733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede078d3-2454-4e28-baf7-fb9a25829f3a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.088354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.088307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxdv\" (UniqueName: \"kubernetes.io/projected/ede078d3-2454-4e28-baf7-fb9a25829f3a-kube-api-access-vjxdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb\" (UID: \"ede078d3-2454-4e28-baf7-fb9a25829f3a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.181145 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.181045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:13.460635 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.460605 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb"] Sep 18 22:12:13.462881 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:12:13.462842 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede078d3_2454_4e28_baf7_fb9a25829f3a.slice/crio-8d153e74822ff6ff42297eb9b65c65132b36ff76f17b875cc3e204db29d269f6 WatchSource:0}: Error finding container 8d153e74822ff6ff42297eb9b65c65132b36ff76f17b875cc3e204db29d269f6: Status 404 returned error can't find the container with id 8d153e74822ff6ff42297eb9b65c65132b36ff76f17b875cc3e204db29d269f6 Sep 18 22:12:13.916865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.916822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" event={"ID":"ede078d3-2454-4e28-baf7-fb9a25829f3a","Type":"ContainerStarted","Data":"8d5b0c8a0625b66cd68ba80fdf68f1197d7a73e899419da8b8226e235fa978bc"} Sep 18 22:12:13.917265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.916889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" event={"ID":"ede078d3-2454-4e28-baf7-fb9a25829f3a","Type":"ContainerStarted","Data":"8d153e74822ff6ff42297eb9b65c65132b36ff76f17b875cc3e204db29d269f6"} Sep 18 22:12:13.943448 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:13.943381 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" podStartSLOduration=1.758663157 podStartE2EDuration="1.943359724s" totalImagesPullingTime="184.696567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:12:13.941768267 +0000 UTC m=+719.613266250" watchObservedRunningTime="2026-09-18 22:12:13.943359724 +0000 UTC m=+719.614857695" Sep 18 22:12:14.316435 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:14.316399 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:12:23.182218 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:23.182169 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:23.182678 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:23.182237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:23.195086 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:23.195056 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:23.977250 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:23.977215 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb" Sep 18 22:12:28.462255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:28.462216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:12:30.226347 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:30.226309 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:12:35.895063 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:35.895030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:12:38.258642 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.258601 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx"] Sep 18 22:12:38.263204 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.263179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.268120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.268099 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 18 22:12:38.280348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.280315 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx"] Sep 18 22:12:38.308634 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.308634 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvtn\" (UniqueName: \"kubernetes.io/projected/d7c0d48f-bba4-4ff1-965c-7340d07c8102-kube-api-access-fpvtn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.308878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.308878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.308878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.308878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.308875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.345621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.345578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:12:38.410193 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvtn\" (UniqueName: \"kubernetes.io/projected/d7c0d48f-bba4-4ff1-965c-7340d07c8102-kube-api-access-fpvtn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410734 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.410781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.410761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.412983 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.412957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7c0d48f-bba4-4ff1-965c-7340d07c8102-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.413098 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.413054 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c0d48f-bba4-4ff1-965c-7340d07c8102-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.421985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.421958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvtn\" (UniqueName: \"kubernetes.io/projected/d7c0d48f-bba4-4ff1-965c-7340d07c8102-kube-api-access-fpvtn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx\" (UID: \"d7c0d48f-bba4-4ff1-965c-7340d07c8102\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.572749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.572652 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:38.856965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:38.856936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx"] Sep 18 22:12:38.859072 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:12:38.859041 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7c0d48f_bba4_4ff1_965c_7340d07c8102.slice/crio-e2d9dde324cd6d33691fe4dbc7b94d3518489f38d749a1df74fafefc158551f9 WatchSource:0}: Error finding container e2d9dde324cd6d33691fe4dbc7b94d3518489f38d749a1df74fafefc158551f9: Status 404 returned error can't find the container with id e2d9dde324cd6d33691fe4dbc7b94d3518489f38d749a1df74fafefc158551f9 Sep 18 22:12:39.038187 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:39.038154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" event={"ID":"d7c0d48f-bba4-4ff1-965c-7340d07c8102","Type":"ContainerStarted","Data":"e2d9dde324cd6d33691fe4dbc7b94d3518489f38d749a1df74fafefc158551f9"} Sep 18 22:12:40.044284 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:40.044248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" event={"ID":"d7c0d48f-bba4-4ff1-965c-7340d07c8102","Type":"ContainerStarted","Data":"c3bcfc51dec7702891b5a95f9a208102ef0eb43babcca8f154fcd2dfb515b237"} Sep 18 22:12:40.068337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:40.068274 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" podStartSLOduration=1.886893135 podStartE2EDuration="2.068255259s" totalImagesPullingTime="181.362124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:12:40.065711892 +0000 UTC m=+745.737209861" watchObservedRunningTime="2026-09-18 22:12:40.068255259 +0000 UTC m=+745.739753228" Sep 18 22:12:40.515334 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:40.515297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:12:48.573749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:48.573698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:48.573749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:48.573762 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:48.586163 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:48.586133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:49.099823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:49.099757 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx" Sep 18 22:12:58.814253 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:12:58.814209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:37.957615 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:37.957526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:38.085839 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:38.085777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:39.444865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:39.444828 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:39.607994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:39.607961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:54.924958 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:54.924909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.031095 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.031063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.276667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.276626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.411447 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.411410 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.575363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.575267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.719735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.719702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:55.910621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:55.910500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.100257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.100223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.239494 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.239460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.360280 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.360230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.500409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.500307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.867827 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.867726 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:56.942326 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.942291 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7651-0a63d7"] Sep 18 22:13:56.945596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.945572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:13:56.948628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.948605 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t97qt\"" Sep 18 22:13:56.956690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:56.956655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7651-0a63d7"] Sep 18 22:13:57.023597 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:57.023564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fln\" (UniqueName: \"kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln\") pod \"test-curl-7651-0a63d7\" (UID: \"75eb7208-8b65-492b-b57a-94cc0f8c26fc\") " pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:13:57.125039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:57.124942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fln\" (UniqueName: \"kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln\") pod \"test-curl-7651-0a63d7\" (UID: \"75eb7208-8b65-492b-b57a-94cc0f8c26fc\") " pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:13:57.137568 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:57.137534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fln\" (UniqueName: \"kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln\") pod \"test-curl-7651-0a63d7\" (UID: \"75eb7208-8b65-492b-b57a-94cc0f8c26fc\") " pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:13:57.257012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:57.256971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:13:57.535804 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:57.535750 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7651-0a63d7"] Sep 18 22:13:57.539099 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:13:57.539066 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75eb7208_8b65_492b_b57a_94cc0f8c26fc.slice/crio-f26d2e3c3d993eda7eeec540e88080da78f8dd923883855375b6478dbe97fbf6 WatchSource:0}: Error finding container f26d2e3c3d993eda7eeec540e88080da78f8dd923883855375b6478dbe97fbf6: Status 404 returned error can't find the container with id f26d2e3c3d993eda7eeec540e88080da78f8dd923883855375b6478dbe97fbf6 Sep 18 22:13:58.427839 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:58.427772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7651-0a63d7" event={"ID":"75eb7208-8b65-492b-b57a-94cc0f8c26fc","Type":"ContainerStarted","Data":"f26d2e3c3d993eda7eeec540e88080da78f8dd923883855375b6478dbe97fbf6"} Sep 18 22:13:58.436737 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:58.436700 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:58.558140 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:58.558104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:58.671403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:58.671338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:59.133541 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:59.133501 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:13:59.444683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:59.444647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:13:59.627545 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:13:59.627517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:00.078124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.078090 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:14:00.081724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.081698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.084144 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.084123 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3d28c1-serving-cert\"" Sep 18 22:14:00.112965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.112932 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:14:00.154916 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.154879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwbl\" (UniqueName: \"kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.155090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.154960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.155090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.154994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.155090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.155026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.183918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.183888 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:00.256064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.256032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.256064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.256071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.256309 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.256100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.256309 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.256168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwbl\" (UniqueName: \"kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.256309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:00.256212 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-serving-cert: secret "maas-api-e2e-inf-a-3d28c1-serving-cert" not found Sep 18 22:14:00.256309 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:00.256296 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls podName:ac59a79b-812f-4b62-b0f8-18e4e2132866 nodeName:}" failed. No retries permitted until 2026-09-18 22:14:00.756272166 +0000 UTC m=+826.427770130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls") pod "maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" (UID: "ac59a79b-812f-4b62-b0f8-18e4e2132866") : secret "maas-api-e2e-inf-a-3d28c1-serving-cert" not found Sep 18 22:14:00.256745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.256724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.258906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.258888 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.271343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.271315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwbl\" (UniqueName: \"kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.276329 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.276296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:00.441506 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.441473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7651-0a63d7" event={"ID":"75eb7208-8b65-492b-b57a-94cc0f8c26fc","Type":"ContainerStarted","Data":"82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3"} Sep 18 22:14:00.465191 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.465135 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7651-0a63d7" podStartSLOduration=2.401557935 podStartE2EDuration="4.465117587s" totalImagesPullingTime="2.063559652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:00.464152112 +0000 UTC m=+826.135650081" watchObservedRunningTime="2026-09-18 22:14:00.465117587 +0000 UTC m=+826.136615589" Sep 18 22:14:00.704236 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.704142 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:00.762026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.761981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.764739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.764708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:00.995130 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:00.995089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:01.109755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.109719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:01.240840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.240717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:01.251053 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.251020 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7651-0a63d7"] Sep 18 22:14:01.257547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.257497 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7651-0a63d7"] Sep 18 22:14:01.289928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.289899 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:14:01.292014 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:01.291987 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac59a79b_812f_4b62_b0f8_18e4e2132866.slice/crio-77e7159838ca124f534aea58da9723cf863fbdb24398661074cff80ac0068e51 WatchSource:0}: Error finding container 77e7159838ca124f534aea58da9723cf863fbdb24398661074cff80ac0068e51: Status 404 returned error can't find the container with id 77e7159838ca124f534aea58da9723cf863fbdb24398661074cff80ac0068e51 Sep 18 22:14:01.448191 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.448152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" event={"ID":"ac59a79b-812f-4b62-b0f8-18e4e2132866","Type":"ContainerStarted","Data":"77e7159838ca124f534aea58da9723cf863fbdb24398661074cff80ac0068e51"} Sep 18 22:14:01.448387 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:01.448369 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7651-0a63d7" container="test-curl-7651-0a63d7" Sep 18 22:14:01.803182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:01.803149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:02.454356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:02.454315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" event={"ID":"ac59a79b-812f-4b62-b0f8-18e4e2132866","Type":"ContainerStarted","Data":"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f"} Sep 18 22:14:02.454567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:02.454398 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:02.454567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:02.454456 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7651-0a63d7" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" containerName="test-curl-7651-0a63d7" containerID="cri-o://82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3" gracePeriod=2 Sep 18 22:14:02.460723 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:02.460686 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7651-0a63d7\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:14:02.490341 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:02.490293 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7651-0a63d7\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:14:02.490528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:02.490485 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" podStartSLOduration=2.128146934 podStartE2EDuration="2.490467899s" totalImagesPullingTime="362.320965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:02.486756672 +0000 UTC m=+828.158254642" watchObservedRunningTime="2026-09-18 22:14:02.490467899 +0000 UTC m=+828.161965871" Sep 18 22:14:03.500587 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.500554 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:14:03.500969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.500958 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" containerName="test-curl-7651-0a63d7" Sep 18 22:14:03.501010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.500973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" containerName="test-curl-7651-0a63d7" Sep 18 22:14:03.501079 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.501068 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" containerName="test-curl-7651-0a63d7" Sep 18 22:14:03.504698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.504677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.520600 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:03.520562 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.521761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.521744 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-59b357-serving-cert\"" Sep 18 22:14:03.577866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.577827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:14:03.588417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.588391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.588515 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.588428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.588601 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.588523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg46d\" (UniqueName: \"kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.588601 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.588592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.651153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.651105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:03.689567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.689528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg46d\" (UniqueName: \"kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.689770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.689590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.689770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.689617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.689770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.689638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.689770 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:03.689756 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-serving-cert: secret "maas-api-e2e-worker-w4-59b357-serving-cert" not found Sep 18 22:14:03.690040 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:03.689867 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls podName:f76c7554-c4ad-4ca4-8fee-be5948f81176 nodeName:}" failed. No retries permitted until 2026-09-18 22:14:04.189844371 +0000 UTC m=+829.861342324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls") pod "maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" (UID: "f76c7554-c4ad-4ca4-8fee-be5948f81176") : secret "maas-api-e2e-worker-w4-59b357-serving-cert" not found Sep 18 22:14:03.690301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.690280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.692450 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.692424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.705291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.705255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg46d\" (UniqueName: \"kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:03.793391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:03.793305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:04.048922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.048838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:04.195503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.195461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:04.198266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.198235 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:04.415107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.415016 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:04.532202 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.532135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:04.632635 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.632602 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:14:04.642022 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.641996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:04.699602 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.699575 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6fln\" (UniqueName: \"kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln\") pod \"75eb7208-8b65-492b-b57a-94cc0f8c26fc\" (UID: \"75eb7208-8b65-492b-b57a-94cc0f8c26fc\") " Sep 18 22:14:04.702608 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.702566 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln" (OuterVolumeSpecName: "kube-api-access-c6fln") pod "75eb7208-8b65-492b-b57a-94cc0f8c26fc" (UID: "75eb7208-8b65-492b-b57a-94cc0f8c26fc"). InnerVolumeSpecName "kube-api-access-c6fln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:14:04.703020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.702900 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:14:04.705834 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:04.705805 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76c7554_c4ad_4ca4_8fee_be5948f81176.slice/crio-4ef6b5021a3063d41e3a6b4afd93371c7179dc4eb5417eeb540d32bbdd1478b9 WatchSource:0}: Error finding container 4ef6b5021a3063d41e3a6b4afd93371c7179dc4eb5417eeb540d32bbdd1478b9: Status 404 returned error can't find the container with id 4ef6b5021a3063d41e3a6b4afd93371c7179dc4eb5417eeb540d32bbdd1478b9 Sep 18 22:14:04.800981 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.800946 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fln\" (UniqueName: \"kubernetes.io/projected/75eb7208-8b65-492b-b57a-94cc0f8c26fc-kube-api-access-c6fln\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:14:04.854028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.853984 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" path="/var/lib/kubelet/pods/75eb7208-8b65-492b-b57a-94cc0f8c26fc/volumes" Sep 18 22:14:04.919241 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:04.919206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:05.088000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.087967 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:14:05.091533 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.091512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.094621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.094592 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a13357-serving-cert\"" Sep 18 22:14:05.113620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.113578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:14:05.204908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.204870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.205087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.204928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.205087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.205041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.205087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.205069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgrt\" (UniqueName: \"kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.261990 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.261954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:05.306219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.306128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.306219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.306180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.306748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.306222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.306748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.306237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgrt\" (UniqueName: \"kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.306748 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:05.306300 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-serving-cert: secret "maas-api-e2e-worker-w0-a13357-serving-cert" not found Sep 18 22:14:05.306748 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:05.306389 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls podName:14b561d3-110b-4ece-9651-e85945ef9aa3 nodeName:}" failed. No retries permitted until 2026-09-18 22:14:05.806363855 +0000 UTC m=+831.477861817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls") pod "maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" (UID: "14b561d3-110b-4ece-9651-e85945ef9aa3") : secret "maas-api-e2e-worker-w0-a13357-serving-cert" not found Sep 18 22:14:05.307132 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.307108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.309165 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.309143 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.325754 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.325723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgrt\" (UniqueName: \"kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.379135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.379101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:05.473525 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.473486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" event={"ID":"f76c7554-c4ad-4ca4-8fee-be5948f81176","Type":"ContainerStarted","Data":"6f8faf59d24d2456abb60806366908024d03d1ec5b84d5a3df58e029531a7316"} Sep 18 22:14:05.473745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.473647 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:05.473745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.473677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" event={"ID":"f76c7554-c4ad-4ca4-8fee-be5948f81176","Type":"ContainerStarted","Data":"4ef6b5021a3063d41e3a6b4afd93371c7179dc4eb5417eeb540d32bbdd1478b9"} Sep 18 22:14:05.474893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.474860 2577 generic.go:356] "Generic (PLEG): container finished" podID="75eb7208-8b65-492b-b57a-94cc0f8c26fc" containerID="82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3" exitCode=137 Sep 18 22:14:05.474989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.474914 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7651-0a63d7" Sep 18 22:14:05.474989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.474921 2577 scope.go:117] "RemoveContainer" containerID="82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3" Sep 18 22:14:05.484138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.484118 2577 scope.go:117] "RemoveContainer" containerID="82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3" Sep 18 22:14:05.484542 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:05.484523 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3\": container with ID starting with 82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3 not found: ID does not exist" containerID="82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3" Sep 18 22:14:05.484583 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.484551 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3"} err="failed to get container status \"82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3\": rpc error: code = NotFound desc = could not find container \"82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3\": container with ID starting with 82fe4b0db4239a186554fc10805edf2049eb0ae2438270873dca8d50afc3c7e3 not found: ID does not exist" Sep 18 22:14:05.502283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.502228 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" podStartSLOduration=2.186707952 podStartE2EDuration="2.502210635s" totalImagesPullingTime="315.502683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:05.501699996 +0000 UTC m=+831.173197965" watchObservedRunningTime="2026-09-18 22:14:05.502210635 +0000 UTC m=+831.173708605" Sep 18 22:14:05.537588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.537552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:05.639287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.639206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:05.812389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.812347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.815093 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.815063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:05.985230 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:05.985193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:06.003726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:06.003686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:06.076162 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:06.076136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:06.293370 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:06.293340 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:14:06.296100 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:06.296067 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b561d3_110b_4ece_9651_e85945ef9aa3.slice/crio-c49391bf0c314b8e6441fd5864e746e10edcd88f8447cff94a46cbbbd5b3a611 WatchSource:0}: Error finding container c49391bf0c314b8e6441fd5864e746e10edcd88f8447cff94a46cbbbd5b3a611: Status 404 returned error can't find the container with id c49391bf0c314b8e6441fd5864e746e10edcd88f8447cff94a46cbbbd5b3a611 Sep 18 22:14:06.482124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:06.482087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" event={"ID":"14b561d3-110b-4ece-9651-e85945ef9aa3","Type":"ContainerStarted","Data":"c49391bf0c314b8e6441fd5864e746e10edcd88f8447cff94a46cbbbd5b3a611"} Sep 18 22:14:06.527591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:06.527562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:07.113829 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.113778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:07.233363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.233332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:07.404423 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.404333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:07.488664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.488628 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" event={"ID":"14b561d3-110b-4ece-9651-e85945ef9aa3","Type":"ContainerStarted","Data":"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56"} Sep 18 22:14:07.488891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.488710 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:07.509321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.509249 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" podStartSLOduration=2.152343252 podStartE2EDuration="2.509233349s" totalImagesPullingTime="356.890097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:07.507132847 +0000 UTC m=+833.178630817" watchObservedRunningTime="2026-09-18 22:14:07.509233349 +0000 UTC m=+833.180731316" Sep 18 22:14:07.793674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.793642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:07.967521 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:07.967489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:08.465073 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:08.465031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:14:08.602554 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:08.602523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:08.792670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:08.792585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:09.460551 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:09.460517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:10.757836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.757801 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:14:10.761417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.761397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.765039 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.765016 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0ffe43-serving-cert\"" Sep 18 22:14:10.781776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.781743 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:14:10.860755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.860725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.860987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.860778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.860987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.860933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.860987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.860967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxrb\" (UniqueName: \"kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.861155 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.861026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:10.962400 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.962366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.962400 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.962408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxrb\" (UniqueName: \"kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.962690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.962450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.962690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.962468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.962690 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:10.962604 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-serving-cert: secret "maas-api-e2e-autores-0ffe43-serving-cert" not found Sep 18 22:14:10.962690 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:10.962673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls podName:eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db nodeName:}" failed. No retries permitted until 2026-09-18 22:14:11.462650117 +0000 UTC m=+837.134148065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls") pod "maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" (UID: "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db") : secret "maas-api-e2e-autores-0ffe43-serving-cert" not found Sep 18 22:14:10.963228 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.963210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.965175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.965149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.976618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.976590 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxrb\" (UniqueName: \"kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:10.988009 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:10.987986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:11.118133 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.118044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:11.335653 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.335620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:11.468548 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.468515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:11.471305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.471274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") pod \"maas-api-e2e-autores-0ffe43-8db8f6887-89bnx\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:11.487704 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.487678 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:14:11.607756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.607727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:11.672389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.672349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:11.967916 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:11.967892 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:14:11.970174 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:11.970141 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6b88b6_68dd_4ea5_a4eb_4ff030fbe4db.slice/crio-08b8c54d81924dc7375d15d2176ca8c19c54ef028a45adf04bfac3f12bd9533c WatchSource:0}: Error finding container 08b8c54d81924dc7375d15d2176ca8c19c54ef028a45adf04bfac3f12bd9533c: Status 404 returned error can't find the container with id 08b8c54d81924dc7375d15d2176ca8c19c54ef028a45adf04bfac3f12bd9533c Sep 18 22:14:12.471286 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.471246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:12.519301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.519270 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" event={"ID":"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db","Type":"ContainerStarted","Data":"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb"} Sep 18 22:14:12.519451 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.519319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" event={"ID":"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db","Type":"ContainerStarted","Data":"08b8c54d81924dc7375d15d2176ca8c19c54ef028a45adf04bfac3f12bd9533c"} Sep 18 22:14:12.519451 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.519364 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:12.557385 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.557276 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" podStartSLOduration=2.24553407 podStartE2EDuration="2.557252494s" totalImagesPullingTime="311.718424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:12.554897522 +0000 UTC m=+838.226395491" watchObservedRunningTime="2026-09-18 22:14:12.557252494 +0000 UTC m=+838.228750464" Sep 18 22:14:12.592136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:12.592102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:13.402060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:13.402027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:13.501467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:13.501436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:14:14.574508 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:14.574477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:15.525499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:15.525468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:15.887081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:15.886985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.077183 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.077148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.208305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.208269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.306392 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.306361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.673090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.673008 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:16.685537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.685501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.689293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.689265 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-82b6dcec-serving-cert\"" Sep 18 22:14:16.714891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.714850 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:16.784252 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.784219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.825257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.825211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.825257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.825256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.825466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.825392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.825466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.825447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgdj\" (UniqueName: \"kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.877532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.877497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:16.926937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.926832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.926937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.926884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.926937 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:16.926913 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-serving-cert: secret "maas-api-e2e-ait-82b6dcec-serving-cert" not found Sep 18 22:14:16.927510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.926938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.927510 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:16.926980 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls podName:eb1fc527-b98a-4c88-8495-367cb3850364 nodeName:}" failed. No retries permitted until 2026-09-18 22:14:17.426957115 +0000 UTC m=+843.098455069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls") pod "maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" (UID: "eb1fc527-b98a-4c88-8495-367cb3850364") : secret "maas-api-e2e-ait-82b6dcec-serving-cert" not found Sep 18 22:14:16.927510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.927016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgdj\" (UniqueName: \"kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.927697 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.927676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.929699 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.929674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:16.937835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:16.937782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgdj\" (UniqueName: \"kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:17.184736 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.184702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:17.432471 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.432434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:17.435206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.435123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") pod \"maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:17.596918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.596855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:17.895000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.894907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:17.921422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:17.921283 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:17.924063 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:17.924028 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1fc527_b98a_4c88_8495_367cb3850364.slice/crio-552b172a6c179c58b559fcd42ce5aef45fef2d262f900748e0fe09a522c0e27b WatchSource:0}: Error finding container 552b172a6c179c58b559fcd42ce5aef45fef2d262f900748e0fe09a522c0e27b: Status 404 returned error can't find the container with id 552b172a6c179c58b559fcd42ce5aef45fef2d262f900748e0fe09a522c0e27b Sep 18 22:14:18.139112 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.139079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.255986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.255954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.354632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.354582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.511403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.511280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.531704 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.531674 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:14:18.555535 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.555491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" event={"ID":"eb1fc527-b98a-4c88-8495-367cb3850364","Type":"ContainerStarted","Data":"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b"} Sep 18 22:14:18.555818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.555592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" event={"ID":"eb1fc527-b98a-4c88-8495-367cb3850364","Type":"ContainerStarted","Data":"552b172a6c179c58b559fcd42ce5aef45fef2d262f900748e0fe09a522c0e27b"} Sep 18 22:14:18.555818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.555628 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:18.587147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.587084 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" podStartSLOduration=2.292449384 podStartE2EDuration="2.58705763s" totalImagesPullingTime="294.608246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:18.584166295 +0000 UTC m=+844.255664269" watchObservedRunningTime="2026-09-18 22:14:18.58705763 +0000 UTC m=+844.258555602" Sep 18 22:14:18.610546 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.610515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.726720 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.726678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:18.855621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:18.855524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:19.066645 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:19.066582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:19.208429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:19.208395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:20.179557 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:20.179524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:20.712775 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:20.712739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:14:20.900149 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:20.900103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:22.151381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:22.151346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:23.223455 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:23.223420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:23.495117 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:23.495016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:23.874152 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:23.874058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:24.347526 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:24.347492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:24.566508 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:24.566477 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:24.850198 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:24.850164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:25.019512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:25.019478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:25.412931 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:25.412838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:26.953069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:26.953042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:28.292752 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:28.292706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:29.408115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.408076 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:14:29.416391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.416363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.419285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.419258 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab559b-serving-cert\"" Sep 18 22:14:29.430006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.429962 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:14:29.447599 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.447567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:29.457858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.457820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pkv\" (UniqueName: \"kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.458028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.457875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.458028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.457947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.458028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.457981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.546273 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.546229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:29.558642 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.558597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.558761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.558682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.558761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.558721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.558862 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.558780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pkv\" (UniqueName: \"kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.558917 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:29.558860 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-serving-cert: secret "maas-api-e2e-inf-b-ab559b-serving-cert" not found Sep 18 22:14:29.558969 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:29.558924 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls podName:105b066f-c994-4c45-8f14-fee0d2622291 nodeName:}" failed. No retries permitted until 2026-09-18 22:14:30.058906879 +0000 UTC m=+855.730404827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls") pod "maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" (UID: "105b066f-c994-4c45-8f14-fee0d2622291") : secret "maas-api-e2e-inf-b-ab559b-serving-cert" not found Sep 18 22:14:29.559432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.559412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.561412 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.561381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.567750 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.567721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pkv\" (UniqueName: \"kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:29.778959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:29.778927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:30.063712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.063602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:30.066250 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.066218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:30.066618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.066598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:30.167092 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.167058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:30.328822 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.328684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:30.608041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.607956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:14:30.610418 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:30.610385 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b066f_c994_4c45_8f14_fee0d2622291.slice/crio-d4ead0d96c46874d3116769d1fa5edaf3e9cf7ad83d340c9b10dddd8eb1ab89b WatchSource:0}: Error finding container d4ead0d96c46874d3116769d1fa5edaf3e9cf7ad83d340c9b10dddd8eb1ab89b: Status 404 returned error can't find the container with id d4ead0d96c46874d3116769d1fa5edaf3e9cf7ad83d340c9b10dddd8eb1ab89b Sep 18 22:14:30.619693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.619650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" event={"ID":"105b066f-c994-4c45-8f14-fee0d2622291","Type":"ContainerStarted","Data":"d4ead0d96c46874d3116769d1fa5edaf3e9cf7ad83d340c9b10dddd8eb1ab89b"} Sep 18 22:14:30.732670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:30.732638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:31.627373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:31.627342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" event={"ID":"105b066f-c994-4c45-8f14-fee0d2622291","Type":"ContainerStarted","Data":"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7"} Sep 18 22:14:31.627821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:31.627479 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:31.646491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:31.646430 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" podStartSLOduration=2.243673902 podStartE2EDuration="2.646412089s" totalImagesPullingTime="402.738187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:31.645235211 +0000 UTC m=+857.316733183" watchObservedRunningTime="2026-09-18 22:14:31.646412089 +0000 UTC m=+857.317910058" Sep 18 22:14:31.726952 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:31.726917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:32.743539 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:32.743504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:33.704664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:33.704624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:34.654188 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:34.654149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:35.574289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:35.574250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:36.414118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:36.414080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:37.349402 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:37.349362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:37.638684 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:37.638601 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:14:38.509001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:38.508937 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.021108 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.021072 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:42.021527 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.021357 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" podUID="eb1fc527-b98a-4c88-8495-367cb3850364" containerName="maas-api" containerID="cri-o://ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b" gracePeriod=30 Sep 18 22:14:42.062414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.062382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.153717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.153681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.230330 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.230249 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.291593 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.291564 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:42.321301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.321270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.388126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.388093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") pod \"eb1fc527-b98a-4c88-8495-367cb3850364\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " Sep 18 22:14:42.388343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.388176 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs\") pod \"eb1fc527-b98a-4c88-8495-367cb3850364\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " Sep 18 22:14:42.388343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.388207 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca\") pod \"eb1fc527-b98a-4c88-8495-367cb3850364\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " Sep 18 22:14:42.388343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.388231 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvgdj\" (UniqueName: \"kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj\") pod \"eb1fc527-b98a-4c88-8495-367cb3850364\" (UID: \"eb1fc527-b98a-4c88-8495-367cb3850364\") " Sep 18 22:14:42.389283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.389248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb1fc527-b98a-4c88-8495-367cb3850364" (UID: "eb1fc527-b98a-4c88-8495-367cb3850364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:14:42.390823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.390775 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb1fc527-b98a-4c88-8495-367cb3850364" (UID: "eb1fc527-b98a-4c88-8495-367cb3850364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:14:42.391274 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.391251 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb1fc527-b98a-4c88-8495-367cb3850364" (UID: "eb1fc527-b98a-4c88-8495-367cb3850364"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:14:42.391363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.391340 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj" (OuterVolumeSpecName: "kube-api-access-lvgdj") pod "eb1fc527-b98a-4c88-8495-367cb3850364" (UID: "eb1fc527-b98a-4c88-8495-367cb3850364"). InnerVolumeSpecName "kube-api-access-lvgdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:14:42.490017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.489979 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:14:42.490017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.490016 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1fc527-b98a-4c88-8495-367cb3850364-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:14:42.490017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.490031 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgdj\" (UniqueName: \"kubernetes.io/projected/eb1fc527-b98a-4c88-8495-367cb3850364-kube-api-access-lvgdj\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:14:42.490289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.490043 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1fc527-b98a-4c88-8495-367cb3850364-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:14:42.596864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.596732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.691572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.691533 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb1fc527-b98a-4c88-8495-367cb3850364" containerID="ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b" exitCode=0 Sep 18 22:14:42.691745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.691583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" event={"ID":"eb1fc527-b98a-4c88-8495-367cb3850364","Type":"ContainerDied","Data":"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b"} Sep 18 22:14:42.691745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.691669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" event={"ID":"eb1fc527-b98a-4c88-8495-367cb3850364","Type":"ContainerDied","Data":"552b172a6c179c58b559fcd42ce5aef45fef2d262f900748e0fe09a522c0e27b"} Sep 18 22:14:42.691745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.691604 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb" Sep 18 22:14:42.691745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.691699 2577 scope.go:117] "RemoveContainer" containerID="ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b" Sep 18 22:14:42.702060 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.702031 2577 scope.go:117] "RemoveContainer" containerID="ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b" Sep 18 22:14:42.702408 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:14:42.702385 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b\": container with ID starting with ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b not found: ID does not exist" containerID="ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b" Sep 18 22:14:42.702510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.702417 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b"} err="failed to get container status \"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b\": rpc error: code = NotFound desc = could not find container \"ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b\": container with ID starting with ea97f2374927bb0a3b539e2705d49c4bb70dc79d2607b274b92d54b791f14a3b not found: ID does not exist" Sep 18 22:14:42.730930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.730887 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:42.743819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.743762 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:42.764425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.764387 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b6dcec-7bd474bdcc-n5dhb"] Sep 18 22:14:42.848751 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.848671 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1fc527-b98a-4c88-8495-367cb3850364" path="/var/lib/kubelet/pods/eb1fc527-b98a-4c88-8495-367cb3850364/volumes" Sep 18 22:14:42.958440 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:42.958396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:43.243428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:43.243397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:43.420842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:43.420811 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:43.759696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:43.759661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:44.513761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:44.513728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:44.928055 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:44.927966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:46.507808 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:46.507753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:46.792391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:46.792305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:47.895606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:47.895575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:48.094399 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:48.094369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:49.348515 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:49.348485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:50.154687 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:50.154641 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:50.206565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:50.206524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:14:50.360297 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:50.360255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:50.608374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:50.608346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:51.697049 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:51.697005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:51.862783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:51.862749 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:52.035058 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.034966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:52.197052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.197003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:52.907606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.907574 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:14:52.908033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.908018 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1fc527-b98a-4c88-8495-367cb3850364" containerName="maas-api" Sep 18 22:14:52.908033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.908031 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1fc527-b98a-4c88-8495-367cb3850364" containerName="maas-api" Sep 18 22:14:52.908132 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.908121 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1fc527-b98a-4c88-8495-367cb3850364" containerName="maas-api" Sep 18 22:14:52.912748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.912722 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.917478 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.917454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"openshift-service-ca.crt\"" Sep 18 22:14:52.922460 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.922435 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"kube-root-ca.crt\"" Sep 18 22:14:52.923315 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.923295 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"default-dockercfg-mtlw6\"" Sep 18 22:14:52.923315 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.923311 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"simulator-w4-59b357-kserve-self-signed-certs\"" Sep 18 22:14:52.935512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.935467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:14:52.993156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxjg\" (UniqueName: \"kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.993156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.993422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.993422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.993422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:52.993422 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:52.993304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.016372 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.016333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:53.094342 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxjg\" (UniqueName: \"kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.094985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.094959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.096984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.096961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.097345 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.097312 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.128627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.128586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxjg\" (UniqueName: \"kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg\") pod \"simulator-w4-59b357-kserve-644f97ccd-n25n5\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.223266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.223229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:14:53.519800 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.519756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:14:53.522375 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:53.522346 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ecf3cd1_3498_4e6d_91bc_f9cecc022930.slice/crio-6f82b33408472ff46120ed41267d07a2cc96b9d732c11447e3dd5eb376358d5a WatchSource:0}: Error finding container 6f82b33408472ff46120ed41267d07a2cc96b9d732c11447e3dd5eb376358d5a: Status 404 returned error can't find the container with id 6f82b33408472ff46120ed41267d07a2cc96b9d732c11447e3dd5eb376358d5a Sep 18 22:14:53.761332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.761289 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:14:53.766711 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.766673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" event={"ID":"8ecf3cd1-3498-4e6d-91bc-f9cecc022930","Type":"ContainerStarted","Data":"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f"} Sep 18 22:14:53.766711 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.766709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" event={"ID":"8ecf3cd1-3498-4e6d-91bc-f9cecc022930","Type":"ContainerStarted","Data":"6f82b33408472ff46120ed41267d07a2cc96b9d732c11447e3dd5eb376358d5a"} Sep 18 22:14:53.767020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.766844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:53.774673 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.774584 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8ed3d70b-serving-cert\"" Sep 18 22:14:53.841384 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.841354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:14:53.850698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.850634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" podStartSLOduration=1.85061551 podStartE2EDuration="1.85061551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:53.846101622 +0000 UTC m=+879.517599594" watchObservedRunningTime="2026-09-18 22:14:53.85061551 +0000 UTC m=+879.522113555" Sep 18 22:14:53.871861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.871831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:53.903273 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.903238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:53.903490 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.903281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgxg\" (UniqueName: \"kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:53.903490 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.903316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:53.903490 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:53.903382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.005028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.004990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.005479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.005045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgxg\" (UniqueName: \"kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.005479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.005066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.005479 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.005108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.006010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.005979 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.007853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.007827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.008061 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.008044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.030978 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.030908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgxg\" (UniqueName: \"kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg\") pod \"maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.061229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.061197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:54.080902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.080864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:54.246954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.246917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:54.386358 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.386331 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:14:54.387325 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:14:54.387297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af507d3_dd5b_4efd_ba48_8ad52baab3a7.slice/crio-839eb9a19c928f5977e5c39a9b932134b919227daedd40bf0985b4190833a920 WatchSource:0}: Error finding container 839eb9a19c928f5977e5c39a9b932134b919227daedd40bf0985b4190833a920: Status 404 returned error can't find the container with id 839eb9a19c928f5977e5c39a9b932134b919227daedd40bf0985b4190833a920 Sep 18 22:14:54.771321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:54.771290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" event={"ID":"5af507d3-dd5b-4efd-ba48-8ad52baab3a7","Type":"ContainerStarted","Data":"839eb9a19c928f5977e5c39a9b932134b919227daedd40bf0985b4190833a920"} Sep 18 22:14:55.035833 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:55.035714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:55.197580 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:55.197548 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:55.778270 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:55.778229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" event={"ID":"5af507d3-dd5b-4efd-ba48-8ad52baab3a7","Type":"ContainerStarted","Data":"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3"} Sep 18 22:14:55.778463 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:55.778366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:14:55.836716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:55.836655 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" podStartSLOduration=2.508322156 podStartE2EDuration="2.836636071s" totalImagesPullingTime="328.313915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:14:55.835285988 +0000 UTC m=+881.506783959" watchObservedRunningTime="2026-09-18 22:14:55.836636071 +0000 UTC m=+881.508134039" Sep 18 22:14:56.205863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:56.205827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:57.848414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:57.848374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:14:59.906398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:14:59.906366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:00.004943 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:00.004909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:01.788925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:01.788893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:15:03.223691 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:03.223652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:15:03.224157 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:03.223707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:15:03.236911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:03.236883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:15:03.829591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:03.829559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:15:15.717287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:15.717241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:15:17.531341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:17.531306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:18.367114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:18.367078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:19.150449 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:19.150412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:20.321001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:20.320967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:21.110172 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:21.110131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:22.265781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:22.265753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:23.128069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:23.128034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:23.999254 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:23.999182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:25.173239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:25.173200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:26.111012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:26.110974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:27.022169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:27.022127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:27.933406 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:27.933375 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:31.818950 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:31.818913 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:32.562630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.562592 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:15:32.562881 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.562858 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" podUID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" containerName="maas-api" containerID="cri-o://3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3" gracePeriod=30 Sep 18 22:15:32.602056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.602027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:32.717674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.717635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:32.814830 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.814734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:32.824394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.824368 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:15:32.903121 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.903084 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs\") pod \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " Sep 18 22:15:32.903327 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.903146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca\") pod \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " Sep 18 22:15:32.903327 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.903190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xgxg\" (UniqueName: \"kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg\") pod \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " Sep 18 22:15:32.903327 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.903298 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls\") pod \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\" (UID: \"5af507d3-dd5b-4efd-ba48-8ad52baab3a7\") " Sep 18 22:15:32.903552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.903523 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5af507d3-dd5b-4efd-ba48-8ad52baab3a7" (UID: "5af507d3-dd5b-4efd-ba48-8ad52baab3a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:15:32.905574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.905533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg" (OuterVolumeSpecName: "kube-api-access-9xgxg") pod "5af507d3-dd5b-4efd-ba48-8ad52baab3a7" (UID: "5af507d3-dd5b-4efd-ba48-8ad52baab3a7"). InnerVolumeSpecName "kube-api-access-9xgxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:15:32.905574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.905548 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5af507d3-dd5b-4efd-ba48-8ad52baab3a7" (UID: "5af507d3-dd5b-4efd-ba48-8ad52baab3a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:15:32.905574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.905572 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5af507d3-dd5b-4efd-ba48-8ad52baab3a7" (UID: "5af507d3-dd5b-4efd-ba48-8ad52baab3a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:15:32.913883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.913833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:32.983858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.983822 2577 generic.go:356] "Generic (PLEG): container finished" podID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" containerID="3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3" exitCode=0 Sep 18 22:15:32.984064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.983873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" event={"ID":"5af507d3-dd5b-4efd-ba48-8ad52baab3a7","Type":"ContainerDied","Data":"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3"} Sep 18 22:15:32.984064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.983905 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" Sep 18 22:15:32.984064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.983928 2577 scope.go:117] "RemoveContainer" containerID="3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3" Sep 18 22:15:32.984064 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.983917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z" event={"ID":"5af507d3-dd5b-4efd-ba48-8ad52baab3a7","Type":"ContainerDied","Data":"839eb9a19c928f5977e5c39a9b932134b919227daedd40bf0985b4190833a920"} Sep 18 22:15:32.994700 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:32.994669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:33.003362 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.003329 2577 scope.go:117] "RemoveContainer" containerID="3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3" Sep 18 22:15:33.003836 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:15:33.003759 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3\": container with ID starting with 3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3 not found: ID does not exist" containerID="3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3" Sep 18 22:15:33.003974 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.003850 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3"} err="failed to get container status \"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3\": rpc error: code = NotFound desc = could not find container \"3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3\": container with ID starting with 3fa4e988ca03b5994129a0bcb230c3b5f9f86c935a76d040349136f936a9e8c3 not found: ID does not exist" Sep 18 22:15:33.004223 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.004206 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:15:33.004290 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.004224 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:15:33.004290 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.004233 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:15:33.004290 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.004244 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgxg\" (UniqueName: \"kubernetes.io/projected/5af507d3-dd5b-4efd-ba48-8ad52baab3a7-kube-api-access-9xgxg\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:15:33.010728 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.010676 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:15:33.012887 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.012858 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ed3d70b-8f84bb766-sr55z"] Sep 18 22:15:33.129893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.129759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:33.433736 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.433702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:33.556592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.556557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:33.890592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:33.890494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:34.501529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:34.501500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:34.656107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:34.656078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:34.848627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:34.848542 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" path="/var/lib/kubelet/pods/5af507d3-dd5b-4efd-ba48-8ad52baab3a7/volumes" Sep 18 22:15:35.602079 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:35.602045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:36.537867 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:36.537834 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:36.830076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:36.829968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:36.929225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:36.929197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:37.957597 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:37.957561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:38.870342 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:38.870306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:39.624497 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.624458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:15:39.706444 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.706407 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:15:39.706885 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.706872 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" containerName="maas-api" Sep 18 22:15:39.706943 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.706887 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" containerName="maas-api" Sep 18 22:15:39.706997 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.706967 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af507d3-dd5b-4efd-ba48-8ad52baab3a7" containerName="maas-api" Sep 18 22:15:39.712232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.712200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.715497 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.715466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0144e4de-serving-cert\"" Sep 18 22:15:39.726890 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.726856 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:15:39.793567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.793528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:39.873196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.873155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.873196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.873205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrht\" (UniqueName: \"kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.873459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.873267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.873459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.873322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.960844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.960806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:39.974173 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.974146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrht\" (UniqueName: \"kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.974283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.974198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.974283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.974236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.974391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.974347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.974526 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:15:39.974464 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-serving-cert: secret "maas-api-e2e-ait-0144e4de-serving-cert" not found Sep 18 22:15:39.974597 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:15:39.974553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls podName:9b3cc748-9590-4d36-a6ab-19c347c8e2cc nodeName:}" failed. No retries permitted until 2026-09-18 22:15:40.474530828 +0000 UTC m=+926.146028783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls") pod "maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" (UID: "9b3cc748-9590-4d36-a6ab-19c347c8e2cc") : secret "maas-api-e2e-ait-0144e4de-serving-cert" not found Sep 18 22:15:39.975019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.974994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.976981 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.976960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:39.990371 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:39.990338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrht\" (UniqueName: \"kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:40.351710 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.351603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:40.479753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.479711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:40.482559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.482529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") pod \"maas-api-e2e-ait-0144e4de-6cb598795-6kzh6\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:40.624847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.624719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:40.794755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.794717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:40.915871 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.915721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:15:40.918448 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:15:40.918419 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3cc748_9590_4d36_a6ab_19c347c8e2cc.slice/crio-0179f409151070ab5ead1899608db51cf8dd3b5a52d2b86343068651fc68f1a3 WatchSource:0}: Error finding container 0179f409151070ab5ead1899608db51cf8dd3b5a52d2b86343068651fc68f1a3: Status 404 returned error can't find the container with id 0179f409151070ab5ead1899608db51cf8dd3b5a52d2b86343068651fc68f1a3 Sep 18 22:15:40.926489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:40.926457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:41.026282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:41.026247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" event={"ID":"9b3cc748-9590-4d36-a6ab-19c347c8e2cc","Type":"ContainerStarted","Data":"0179f409151070ab5ead1899608db51cf8dd3b5a52d2b86343068651fc68f1a3"} Sep 18 22:15:42.033529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:42.033496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" event={"ID":"9b3cc748-9590-4d36-a6ab-19c347c8e2cc","Type":"ContainerStarted","Data":"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486"} Sep 18 22:15:42.033970 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:42.033570 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:42.081745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:42.081670 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" podStartSLOduration=2.742576379 podStartE2EDuration="3.081653122s" totalImagesPullingTime="339.076743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:15:42.081079439 +0000 UTC m=+927.752577419" watchObservedRunningTime="2026-09-18 22:15:42.081653122 +0000 UTC m=+927.753151091" Sep 18 22:15:42.108854 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:42.108815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:43.458667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:43.458634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:44.675300 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:44.675271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:45.602967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:45.602931 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:48.045308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:48.045276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:15:48.998906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:48.998873 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:15:49.002767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.002749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.006243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.006215 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0ffe43\"/\"openshift-service-ca.crt\"" Sep 18 22:15:49.006243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.006221 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0ffe43\"/\"kube-root-ca.crt\"" Sep 18 22:15:49.006460 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.006222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0ffe43\"/\"default-dockercfg-26h5d\"" Sep 18 22:15:49.006460 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.006222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0ffe43\"/\"autores-model-0ffe43-kserve-self-signed-certs\"" Sep 18 22:15:49.012418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.012387 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:15:49.068405 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.068838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.068838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7qh\" (UniqueName: \"kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.068838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.068838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.068838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.068686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7qh\" (UniqueName: \"kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169982 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.169982 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.169969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.170192 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.170171 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.172396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.172370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.172863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.172846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.179462 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.179431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7qh\" (UniqueName: \"kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh\") pod \"autores-model-0ffe43-kserve-f8c8df4c9-8mg84\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.223184 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.223146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:15:49.313475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.313385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:49.590496 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:49.590464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:15:49.592422 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:15:49.592394 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d7b52e6_193b_4f19_9882_d370f5e7c073.slice/crio-c4ec3b5d800a8370324ff068eb982712471bf4ef06210ebf0817715c859eae57 WatchSource:0}: Error finding container c4ec3b5d800a8370324ff068eb982712471bf4ef06210ebf0817715c859eae57: Status 404 returned error can't find the container with id c4ec3b5d800a8370324ff068eb982712471bf4ef06210ebf0817715c859eae57 Sep 18 22:15:50.076074 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:50.076032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" event={"ID":"1d7b52e6-193b-4f19-9882-d370f5e7c073","Type":"ContainerStarted","Data":"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f"} Sep 18 22:15:50.076509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:50.076083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" event={"ID":"1d7b52e6-193b-4f19-9882-d370f5e7c073","Type":"ContainerStarted","Data":"c4ec3b5d800a8370324ff068eb982712471bf4ef06210ebf0817715c859eae57"} Sep 18 22:15:50.101025 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:50.100968 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" podStartSLOduration=2.100950528 podStartE2EDuration="2.100950528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:15:50.099269984 +0000 UTC m=+935.770767964" watchObservedRunningTime="2026-09-18 22:15:50.100950528 +0000 UTC m=+935.772448558" Sep 18 22:15:59.313652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:59.313599 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:59.313652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:59.313668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:15:59.326368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:15:59.326335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:16:00.139762 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:00.139729 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:16:02.718507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:02.718465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:16:03.322203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:03.322158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:04.834328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:04.834296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:05.909323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:05.909279 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:06.853319 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:06.853274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:08.200126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:08.200088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:09.116919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:09.116885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:09.932738 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:09.932702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:11.388838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:11.388770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:12.000522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:12.000486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:12.759810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:12.759752 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:13.753574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:13.753538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:14.604934 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:14.604904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:18.360903 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:18.360874 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.150985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.150940 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:16:19.151260 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.151212 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" podUID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" containerName="maas-api" containerID="cri-o://82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486" gracePeriod=30 Sep 18 22:16:19.190195 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.190159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.281935 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.281904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.359566 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.359532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.418209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.418137 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:16:19.441558 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.441529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.475017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.474986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca\") pod \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " Sep 18 22:16:19.475199 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.475062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs\") pod \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " Sep 18 22:16:19.475199 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.475126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqrht\" (UniqueName: \"kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht\") pod \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " Sep 18 22:16:19.475324 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.475260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") pod \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\" (UID: \"9b3cc748-9590-4d36-a6ab-19c347c8e2cc\") " Sep 18 22:16:19.475394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.475374 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b3cc748-9590-4d36-a6ab-19c347c8e2cc" (UID: "9b3cc748-9590-4d36-a6ab-19c347c8e2cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:16:19.475614 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.475594 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:19.477535 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.477499 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b3cc748-9590-4d36-a6ab-19c347c8e2cc" (UID: "9b3cc748-9590-4d36-a6ab-19c347c8e2cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:16:19.477655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.477530 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b3cc748-9590-4d36-a6ab-19c347c8e2cc" (UID: "9b3cc748-9590-4d36-a6ab-19c347c8e2cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:16:19.477655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.477553 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht" (OuterVolumeSpecName: "kube-api-access-rqrht") pod "9b3cc748-9590-4d36-a6ab-19c347c8e2cc" (UID: "9b3cc748-9590-4d36-a6ab-19c347c8e2cc"). InnerVolumeSpecName "kube-api-access-rqrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:16:19.514920 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.514876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:19.577033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.576986 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:19.577033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.577017 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:19.577033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.577027 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrht\" (UniqueName: \"kubernetes.io/projected/9b3cc748-9590-4d36-a6ab-19c347c8e2cc-kube-api-access-rqrht\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:19.903395 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:19.903355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:20.234047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.234013 2577 generic.go:356] "Generic (PLEG): container finished" podID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" containerID="82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486" exitCode=0 Sep 18 22:16:20.234227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.234058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" event={"ID":"9b3cc748-9590-4d36-a6ab-19c347c8e2cc","Type":"ContainerDied","Data":"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486"} Sep 18 22:16:20.234227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.234104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" event={"ID":"9b3cc748-9590-4d36-a6ab-19c347c8e2cc","Type":"ContainerDied","Data":"0179f409151070ab5ead1899608db51cf8dd3b5a52d2b86343068651fc68f1a3"} Sep 18 22:16:20.234227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.234127 2577 scope.go:117] "RemoveContainer" containerID="82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486" Sep 18 22:16:20.234227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.234060 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6" Sep 18 22:16:20.244751 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.244731 2577 scope.go:117] "RemoveContainer" containerID="82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486" Sep 18 22:16:20.245204 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:16:20.245185 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486\": container with ID starting with 82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486 not found: ID does not exist" containerID="82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486" Sep 18 22:16:20.245260 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.245214 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486"} err="failed to get container status \"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486\": rpc error: code = NotFound desc = could not find container \"82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486\": container with ID starting with 82ead2765643d7174cb503f92bbcc7b9b1cd1904e560e8bc763491ea6bd78486 not found: ID does not exist" Sep 18 22:16:20.261011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.260965 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:16:20.266129 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.266099 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0144e4de-6cb598795-6kzh6"] Sep 18 22:16:20.637911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.637826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:20.757707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.757675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:20.849210 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:20.849173 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" path="/var/lib/kubelet/pods/9b3cc748-9590-4d36-a6ab-19c347c8e2cc/volumes" Sep 18 22:16:21.051775 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:21.051738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:21.536358 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:21.536325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:21.732569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:21.732542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:21.883689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:21.883610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:23.040912 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:23.040877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:24.019122 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:24.019080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:24.168695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:24.168661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:24.308826 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:24.308701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:25.323748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:25.323707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:16:25.365537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:25.365510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:26.207681 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:26.207642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:27.115582 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:27.115546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:28.018531 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:28.018494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:29.040677 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:29.040626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:29.113182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:29.113148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:29.287350 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:29.287313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:29.696647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:29.696610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:29.793836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:29.793771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:30.220638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:30.220604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:31.012846 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:31.012817 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:32.405036 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:32.405004 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.087628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.087586 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:16:33.127080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.127043 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.210731 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.210694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.277960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.277922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.303061 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.303019 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerName="maas-api" containerID="cri-o://4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb" gracePeriod=30 Sep 18 22:16:33.367281 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.367189 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.440938 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.440906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.583169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.583131 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:16:33.720674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.720523 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs\") pod \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " Sep 18 22:16:33.720929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.720705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca\") pod \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " Sep 18 22:16:33.720929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.720754 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngxrb\" (UniqueName: \"kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb\") pod \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " Sep 18 22:16:33.720929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.720885 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") pod \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\" (UID: \"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db\") " Sep 18 22:16:33.721133 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.721098 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" (UID: "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:16:33.721320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.721300 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:33.723184 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.723140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb" (OuterVolumeSpecName: "kube-api-access-ngxrb") pod "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" (UID: "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db"). InnerVolumeSpecName "kube-api-access-ngxrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:16:33.723308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.723196 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" (UID: "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:16:33.723308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.723208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" (UID: "eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:16:33.728096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.728068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:33.821932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.821894 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:33.821932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.821923 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:33.821932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.821932 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngxrb\" (UniqueName: \"kubernetes.io/projected/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db-kube-api-access-ngxrb\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:33.831944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:33.831911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:34.309643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.309609 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerID="4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb" exitCode=0 Sep 18 22:16:34.309831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.309657 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" Sep 18 22:16:34.309831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.309701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" event={"ID":"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db","Type":"ContainerDied","Data":"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb"} Sep 18 22:16:34.309831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.309755 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" event={"ID":"eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db","Type":"ContainerDied","Data":"08b8c54d81924dc7375d15d2176ca8c19c54ef028a45adf04bfac3f12bd9533c"} Sep 18 22:16:34.309831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.309782 2577 scope.go:117] "RemoveContainer" containerID="4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb" Sep 18 22:16:34.320237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.320219 2577 scope.go:117] "RemoveContainer" containerID="4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb" Sep 18 22:16:34.320614 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:16:34.320594 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb\": container with ID starting with 4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb not found: ID does not exist" containerID="4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb" Sep 18 22:16:34.320665 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.320628 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb"} err="failed to get container status \"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb\": rpc error: code = NotFound desc = could not find container \"4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb\": container with ID starting with 4b0e25bb6aa8549dd9a8a56892daf3c012287e2b16ef9c7e8b03013fbc8b8fcb not found: ID does not exist" Sep 18 22:16:34.334348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.334311 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:16:34.338124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.338093 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx"] Sep 18 22:16:34.720618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.720579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:34.849136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:34.849100 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" path="/var/lib/kubelet/pods/eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db/volumes" Sep 18 22:16:35.650139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:35.650083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:36.526263 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:36.526168 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0ffe43-8db8f6887-89bnx" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.38:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 18 22:16:36.561719 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:36.561683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:37.423293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:37.423253 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:38.406054 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:38.406017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:38.715967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:38.715919 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:16:38.811882 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:38.811847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:39.516394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516360 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:16:39.516823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516760 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" containerName="maas-api" Sep 18 22:16:39.516823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516771 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" containerName="maas-api" Sep 18 22:16:39.516823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516807 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerName="maas-api" Sep 18 22:16:39.516823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516813 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerName="maas-api" Sep 18 22:16:39.516971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516879 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3cc748-9590-4d36-a6ab-19c347c8e2cc" containerName="maas-api" Sep 18 22:16:39.516971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.516886 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb6b88b6-68dd-4ea5-a4eb-4ff030fbe4db" containerName="maas-api" Sep 18 22:16:39.524723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.523393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.527838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.527570 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a13357\"/\"openshift-service-ca.crt\"" Sep 18 22:16:39.527838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.527624 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a13357\"/\"kube-root-ca.crt\"" Sep 18 22:16:39.528094 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.527918 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a13357\"/\"default-dockercfg-9x8h9\"" Sep 18 22:16:39.528094 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.528061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a13357\"/\"simulator-w0-a13357-kserve-self-signed-certs\"" Sep 18 22:16:39.530131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.530106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:16:39.684490 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684453 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.684667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.684667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fs9\" (UniqueName: \"kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.684667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.684667 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.684847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.684706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.714861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.714826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:39.786354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fs9\" (UniqueName: \"kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786766 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786846 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786817 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.786846 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.786836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.789084 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.789059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.789216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.789199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.795427 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.795392 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fs9\" (UniqueName: \"kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9\") pod \"simulator-w0-a13357-kserve-85c7699f7c-wqdpv\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:39.836347 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:39.836304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:40.134977 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:40.134948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:16:40.138065 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:16:40.138027 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f2fcc40_53bc_44c0_b50f_20401de6b336.slice/crio-b35d9ffdbf891631a577316f993d8116e373a2ffc187e81fa47c474e40820f37 WatchSource:0}: Error finding container b35d9ffdbf891631a577316f993d8116e373a2ffc187e81fa47c474e40820f37: Status 404 returned error can't find the container with id b35d9ffdbf891631a577316f993d8116e373a2ffc187e81fa47c474e40820f37 Sep 18 22:16:40.343468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:40.343429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" event={"ID":"1f2fcc40-53bc-44c0-b50f-20401de6b336","Type":"ContainerStarted","Data":"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de"} Sep 18 22:16:40.343468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:40.343474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" event={"ID":"1f2fcc40-53bc-44c0-b50f-20401de6b336","Type":"ContainerStarted","Data":"b35d9ffdbf891631a577316f993d8116e373a2ffc187e81fa47c474e40820f37"} Sep 18 22:16:40.361325 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:40.361263 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" podStartSLOduration=1.361244317 podStartE2EDuration="1.361244317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:16:40.360261955 +0000 UTC m=+986.031759928" watchObservedRunningTime="2026-09-18 22:16:40.361244317 +0000 UTC m=+986.032742286" Sep 18 22:16:49.021297 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:49.021255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:16:49.836823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:49.836754 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:49.837017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:49.836838 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:49.850486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:49.850450 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:50.202216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:50.202177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:50.409296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:50.409269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:16:52.028358 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:52.028318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:52.889090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:52.889038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:53.737366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:53.737282 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" podUID="1d7b52e6-193b-4f19-9882-d370f5e7c073" containerName="main" containerID="cri-o://24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f" gracePeriod=45 Sep 18 22:16:53.927498 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:53.927457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:54.006939 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.006859 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:16:54.021009 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.020969 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021042 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021135 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv7qh\" (UniqueName: \"kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021233 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs\") pod \"1d7b52e6-193b-4f19-9882-d370f5e7c073\" (UID: \"1d7b52e6-193b-4f19-9882-d370f5e7c073\") " Sep 18 22:16:54.021289 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021257 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:16:54.021628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:16:54.021628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021412 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home" (OuterVolumeSpecName: "home") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:16:54.021628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021599 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.021628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021619 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.021888 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.021633 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.024396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.024319 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm" (OuterVolumeSpecName: "dshm") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:16:54.024717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.024683 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh" (OuterVolumeSpecName: "kube-api-access-tv7qh") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "kube-api-access-tv7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:16:54.025131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.025100 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d7b52e6-193b-4f19-9882-d370f5e7c073" (UID: "1d7b52e6-193b-4f19-9882-d370f5e7c073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:16:54.122869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.122822 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7b52e6-193b-4f19-9882-d370f5e7c073-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.122869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.122858 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7b52e6-193b-4f19-9882-d370f5e7c073-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.122869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.122867 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv7qh\" (UniqueName: \"kubernetes.io/projected/1d7b52e6-193b-4f19-9882-d370f5e7c073-kube-api-access-tv7qh\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:16:54.421359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.421271 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d7b52e6-193b-4f19-9882-d370f5e7c073" containerID="24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f" exitCode=0 Sep 18 22:16:54.421359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.421315 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" Sep 18 22:16:54.421567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.421358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" event={"ID":"1d7b52e6-193b-4f19-9882-d370f5e7c073","Type":"ContainerDied","Data":"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f"} Sep 18 22:16:54.421567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.421403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84" event={"ID":"1d7b52e6-193b-4f19-9882-d370f5e7c073","Type":"ContainerDied","Data":"c4ec3b5d800a8370324ff068eb982712471bf4ef06210ebf0817715c859eae57"} Sep 18 22:16:54.421567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.421427 2577 scope.go:117] "RemoveContainer" containerID="24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f" Sep 18 22:16:54.432176 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.432155 2577 scope.go:117] "RemoveContainer" containerID="24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f" Sep 18 22:16:54.432507 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:16:54.432484 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f\": container with ID starting with 24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f not found: ID does not exist" containerID="24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f" Sep 18 22:16:54.432601 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.432514 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f"} err="failed to get container status \"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f\": rpc error: code = NotFound desc = could not find container \"24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f\": container with ID starting with 24a28ef153887cd24ba0867efd753b59f945db5cb95c2d320f1cebf88de8e32f not found: ID does not exist" Sep 18 22:16:54.447163 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.447115 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:16:54.451898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.451858 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0ffe43/autores-model-0ffe43-kserve-f8c8df4c9-8mg84"] Sep 18 22:16:54.850532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.850483 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7b52e6-193b-4f19-9882-d370f5e7c073" path="/var/lib/kubelet/pods/1d7b52e6-193b-4f19-9882-d370f5e7c073/volumes" Sep 18 22:16:54.904936 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:54.904903 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:56.039347 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.039315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:56.916472 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.916434 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:16:56.916921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.916907 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7b52e6-193b-4f19-9882-d370f5e7c073" containerName="main" Sep 18 22:16:56.916974 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.916924 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7b52e6-193b-4f19-9882-d370f5e7c073" containerName="main" Sep 18 22:16:56.917017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.917008 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7b52e6-193b-4f19-9882-d370f5e7c073" containerName="main" Sep 18 22:16:56.921891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.921858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.927055 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.927025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d28c1\"/\"default-dockercfg-5jm4x\"" Sep 18 22:16:56.927245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.927024 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d28c1\"/\"test-model-3d28c1-kserve-self-signed-certs\"" Sep 18 22:16:56.927245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.927024 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d28c1\"/\"kube-root-ca.crt\"" Sep 18 22:16:56.927245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.927025 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d28c1\"/\"openshift-service-ca.crt\"" Sep 18 22:16:56.934276 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.934242 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:16:56.952142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9jt\" (UniqueName: \"kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.952366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.952366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.952366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.952366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.952366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.952326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:56.964949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:56.964917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:57.053197 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9jt\" (UniqueName: \"kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.053946 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.053853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.054071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.054051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.054199 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.054176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.055879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.055856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.056208 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.056187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.065904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.065873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9jt\" (UniqueName: \"kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt\") pod \"test-model-3d28c1-kserve-556db7c499-dbpzs\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.068059 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.068024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:57.234396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.234354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:16:57.525889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:57.525755 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:16:57.530572 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:16:57.530524 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114773e5_5ef2_4f1b_9ccb_f55e204fed49.slice/crio-f8cdfd77ef954a2887b42f2f3a920e9a8b3bafdcf56e6a031e88a76c4062d675 WatchSource:0}: Error finding container f8cdfd77ef954a2887b42f2f3a920e9a8b3bafdcf56e6a031e88a76c4062d675: Status 404 returned error can't find the container with id f8cdfd77ef954a2887b42f2f3a920e9a8b3bafdcf56e6a031e88a76c4062d675 Sep 18 22:16:58.039164 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:58.039126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:58.448719 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:58.448675 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" event={"ID":"114773e5-5ef2-4f1b-9ccb-f55e204fed49","Type":"ContainerStarted","Data":"61d9bbd0f28a040d4aa6172640c4ac740267e31f38e5e737db554344fcb92307"} Sep 18 22:16:58.449149 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:58.448729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" event={"ID":"114773e5-5ef2-4f1b-9ccb-f55e204fed49","Type":"ContainerStarted","Data":"f8cdfd77ef954a2887b42f2f3a920e9a8b3bafdcf56e6a031e88a76c4062d675"} Sep 18 22:16:58.474053 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:58.473991 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" podStartSLOduration=2.473970148 podStartE2EDuration="2.473970148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:16:58.471223287 +0000 UTC m=+1004.142721260" watchObservedRunningTime="2026-09-18 22:16:58.473970148 +0000 UTC m=+1004.145468147" Sep 18 22:16:59.030689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:59.030647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:16:59.806168 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:16:59.806127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:05.683036 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:05.682995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.471107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.471072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.575679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.575635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.673023 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.672992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.782980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.782884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.863847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.863809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:06.947011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:06.946976 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:07.025676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.025640 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:17:07.235202 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.235156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:17:07.235415 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.235237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:17:07.248128 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.248098 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:17:07.352603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.352559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:07.512579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.512494 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:17:07.667031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.666994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:07.806209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:07.806122 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:08.834538 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:08.834501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:09.351829 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:09.351770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:09.971919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:09.971882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:10.918740 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:10.918703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:11.813713 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:11.813676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:15.496228 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:15.496195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:15.604389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:15.604353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:15.709893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:15.709847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:15.806427 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:15.806337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:16.242831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:16.242802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:18.653175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:18.653137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:21.307995 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:21.307954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:21.605272 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:21.605178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:22.002396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:22.002362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:22.465808 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:22.465764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:22.661234 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:22.661201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:26.379082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:26.379048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:26.524098 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:26.524062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:27.233960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:27.233919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:17:27.439666 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:27.439615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:28.502838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:28.502777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:29.435757 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:29.435719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:29.724961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:29.724866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:30.408951 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:30.408912 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:30.508515 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:30.508482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:30.983177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:30.983143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:31.092908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:31.092775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:31.642297 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:31.642260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:32.565066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:32.565032 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:33.536413 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:33.536385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:34.629847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:34.629779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:34.737320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:34.737280 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:17:34.837582 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:34.837525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:35.575814 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:35.575751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:36.685679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:36.685640 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:40.181085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:40.181046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:40.974475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:40.974437 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:17:40.974750 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:40.974712 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" podUID="14b561d3-110b-4ece-9651-e85945ef9aa3" containerName="maas-api" containerID="cri-o://be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56" gracePeriod=30 Sep 18 22:17:41.033186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.033153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:41.232372 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.232343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:41.247216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.247182 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:17:41.319384 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.319351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs\") pod \"14b561d3-110b-4ece-9651-e85945ef9aa3\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " Sep 18 22:17:41.319384 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.319387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsgrt\" (UniqueName: \"kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt\") pod \"14b561d3-110b-4ece-9651-e85945ef9aa3\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " Sep 18 22:17:41.319603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.319458 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca\") pod \"14b561d3-110b-4ece-9651-e85945ef9aa3\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " Sep 18 22:17:41.319603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.319540 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") pod \"14b561d3-110b-4ece-9651-e85945ef9aa3\" (UID: \"14b561d3-110b-4ece-9651-e85945ef9aa3\") " Sep 18 22:17:41.319996 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.319954 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14b561d3-110b-4ece-9651-e85945ef9aa3" (UID: "14b561d3-110b-4ece-9651-e85945ef9aa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:17:41.322012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.321971 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14b561d3-110b-4ece-9651-e85945ef9aa3" (UID: "14b561d3-110b-4ece-9651-e85945ef9aa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:17:41.322411 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.322387 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt" (OuterVolumeSpecName: "kube-api-access-wsgrt") pod "14b561d3-110b-4ece-9651-e85945ef9aa3" (UID: "14b561d3-110b-4ece-9651-e85945ef9aa3"). InnerVolumeSpecName "kube-api-access-wsgrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:17:41.322505 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.322385 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14b561d3-110b-4ece-9651-e85945ef9aa3" (UID: "14b561d3-110b-4ece-9651-e85945ef9aa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:17:41.420686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.420639 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:41.420686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.420677 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b561d3-110b-4ece-9651-e85945ef9aa3-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:41.420686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.420687 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgrt\" (UniqueName: \"kubernetes.io/projected/14b561d3-110b-4ece-9651-e85945ef9aa3-kube-api-access-wsgrt\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:41.420686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.420696 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b561d3-110b-4ece-9651-e85945ef9aa3-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:41.464837 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.464777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:41.696115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.696077 2577 generic.go:356] "Generic (PLEG): container finished" podID="14b561d3-110b-4ece-9651-e85945ef9aa3" containerID="be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56" exitCode=0 Sep 18 22:17:41.696314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.696130 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" Sep 18 22:17:41.696314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.696176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" event={"ID":"14b561d3-110b-4ece-9651-e85945ef9aa3","Type":"ContainerDied","Data":"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56"} Sep 18 22:17:41.696314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.696230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv" event={"ID":"14b561d3-110b-4ece-9651-e85945ef9aa3","Type":"ContainerDied","Data":"c49391bf0c314b8e6441fd5864e746e10edcd88f8447cff94a46cbbbd5b3a611"} Sep 18 22:17:41.696314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.696259 2577 scope.go:117] "RemoveContainer" containerID="be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56" Sep 18 22:17:41.706233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.706214 2577 scope.go:117] "RemoveContainer" containerID="be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56" Sep 18 22:17:41.706602 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:17:41.706581 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56\": container with ID starting with be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56 not found: ID does not exist" containerID="be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56" Sep 18 22:17:41.706683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.706615 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56"} err="failed to get container status \"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56\": rpc error: code = NotFound desc = could not find container \"be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56\": container with ID starting with be12487d7d635adf38dd8cf762432a7edeb3bbc39a3a09df3a1f27a754511f56 not found: ID does not exist" Sep 18 22:17:41.751998 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.751948 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:17:41.782910 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.782874 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a13357-74b84878b4-brlrv"] Sep 18 22:17:41.790381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:41.790348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:42.011396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:42.011308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:42.325420 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:42.325289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:42.455773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:42.455738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:42.848807 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:42.848756 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b561d3-110b-4ece-9651-e85945ef9aa3" path="/var/lib/kubelet/pods/14b561d3-110b-4ece-9651-e85945ef9aa3/volumes" Sep 18 22:17:43.453771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:43.453742 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:44.402081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:44.402024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:45.557375 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:45.557328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:45.994100 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:45.994065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:46.864420 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:46.864387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:46.969916 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:46.969875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:17:48.435781 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:48.435751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:49.719842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:49.719772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:49.758317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:49.758271 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" podUID="1f2fcc40-53bc-44c0-b50f-20401de6b336" containerName="main" containerID="cri-o://4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de" gracePeriod=45 Sep 18 22:17:50.022326 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.022295 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:17:50.113252 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113218 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113252 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113270 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113294 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113360 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7fs9\" (UniqueName: \"kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113543 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache\") pod \"1f2fcc40-53bc-44c0-b50f-20401de6b336\" (UID: \"1f2fcc40-53bc-44c0-b50f-20401de6b336\") " Sep 18 22:17:50.113674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113598 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home" (OuterVolumeSpecName: "home") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:17:50.113674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113610 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:17:50.113868 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113847 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:17:50.113932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113890 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.113932 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.113912 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.115987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.115949 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm" (OuterVolumeSpecName: "dshm") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:17:50.116127 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.116012 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9" (OuterVolumeSpecName: "kube-api-access-s7fs9") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "kube-api-access-s7fs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:17:50.116127 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.116023 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f2fcc40-53bc-44c0-b50f-20401de6b336" (UID: "1f2fcc40-53bc-44c0-b50f-20401de6b336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:17:50.214641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.214603 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fs9\" (UniqueName: \"kubernetes.io/projected/1f2fcc40-53bc-44c0-b50f-20401de6b336-kube-api-access-s7fs9\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.214641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.214636 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.214641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.214647 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2fcc40-53bc-44c0-b50f-20401de6b336-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.214641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.214654 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2fcc40-53bc-44c0-b50f-20401de6b336-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:17:50.746292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.746253 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f2fcc40-53bc-44c0-b50f-20401de6b336" containerID="4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de" exitCode=0 Sep 18 22:17:50.746771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.746301 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" Sep 18 22:17:50.746771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.746329 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" event={"ID":"1f2fcc40-53bc-44c0-b50f-20401de6b336","Type":"ContainerDied","Data":"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de"} Sep 18 22:17:50.746771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.746371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv" event={"ID":"1f2fcc40-53bc-44c0-b50f-20401de6b336","Type":"ContainerDied","Data":"b35d9ffdbf891631a577316f993d8116e373a2ffc187e81fa47c474e40820f37"} Sep 18 22:17:50.746771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.746397 2577 scope.go:117] "RemoveContainer" containerID="4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de" Sep 18 22:17:50.756351 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.756333 2577 scope.go:117] "RemoveContainer" containerID="4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de" Sep 18 22:17:50.756706 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:17:50.756687 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de\": container with ID starting with 4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de not found: ID does not exist" containerID="4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de" Sep 18 22:17:50.756761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.756715 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de"} err="failed to get container status \"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de\": rpc error: code = NotFound desc = could not find container \"4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de\": container with ID starting with 4c16c8c0af21cd64a0097d11503f146d38238e10bec0a75e9631b559443849de not found: ID does not exist" Sep 18 22:17:50.780082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.780045 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:17:50.783662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.783622 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a13357/simulator-w0-a13357-kserve-85c7699f7c-wqdpv"] Sep 18 22:17:50.849235 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:50.849203 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f2fcc40-53bc-44c0-b50f-20401de6b336" path="/var/lib/kubelet/pods/1f2fcc40-53bc-44c0-b50f-20401de6b336/volumes" Sep 18 22:17:51.182641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:51.182612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:51.349564 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:51.349529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:51.476263 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:51.476175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:51.633082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:51.633050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:17:51.748602 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:17:51.748511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:07.301156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.301069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:07.408170 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408133 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:18:07.408567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408554 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b561d3-110b-4ece-9651-e85945ef9aa3" containerName="maas-api" Sep 18 22:18:07.408625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408569 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b561d3-110b-4ece-9651-e85945ef9aa3" containerName="maas-api" Sep 18 22:18:07.408625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408599 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2fcc40-53bc-44c0-b50f-20401de6b336" containerName="main" Sep 18 22:18:07.408625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408604 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2fcc40-53bc-44c0-b50f-20401de6b336" containerName="main" Sep 18 22:18:07.408724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408682 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2fcc40-53bc-44c0-b50f-20401de6b336" containerName="main" Sep 18 22:18:07.408724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.408693 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b561d3-110b-4ece-9651-e85945ef9aa3" containerName="maas-api" Sep 18 22:18:07.413004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.412975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.417361 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.417333 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"premium-w4-59b357-kserve-self-signed-certs\"" Sep 18 22:18:07.425845 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.425779 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:18:07.434657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.434627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:07.492961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.492918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kmz\" (UniqueName: \"kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.492961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.492975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.493213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.493098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.493213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.493124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.493213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.493143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.493213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.493165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594692 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594692 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kmz\" (UniqueName: \"kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594692 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594692 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594684 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.594928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.594835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.597177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.597148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.597248 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.597227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.604332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.604296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kmz\" (UniqueName: \"kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz\") pod \"premium-w4-59b357-kserve-5785b8464c-qkjnv\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:07.722592 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:07.722543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:08.001353 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:08.001322 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:18:08.004261 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:18:08.004230 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1af7e6_2263_4fc1_a739_60e9815d7fec.slice/crio-4558eee109192350da289b3f4f4e334bf32637d2c2e898556c77e66b9c08d2c7 WatchSource:0}: Error finding container 4558eee109192350da289b3f4f4e334bf32637d2c2e898556c77e66b9c08d2c7: Status 404 returned error can't find the container with id 4558eee109192350da289b3f4f4e334bf32637d2c2e898556c77e66b9c08d2c7 Sep 18 22:18:08.848963 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:08.848929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" event={"ID":"ea1af7e6-2263-4fc1-a739-60e9815d7fec","Type":"ContainerStarted","Data":"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836"} Sep 18 22:18:08.849331 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:08.848973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" event={"ID":"ea1af7e6-2263-4fc1-a739-60e9815d7fec","Type":"ContainerStarted","Data":"4558eee109192350da289b3f4f4e334bf32637d2c2e898556c77e66b9c08d2c7"} Sep 18 22:18:08.868080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:08.868026 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" podStartSLOduration=1.868008862 podStartE2EDuration="1.868008862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:08.866941726 +0000 UTC m=+1074.538439695" watchObservedRunningTime="2026-09-18 22:18:08.868008862 +0000 UTC m=+1074.539506832" Sep 18 22:18:12.433213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:12.433167 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:18:14.052762 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:14.052721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:14.184663 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:14.184627 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:15.273081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:15.273050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:15.400655 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:15.400616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:15.500153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:15.500121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:17.722745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:17.722710 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:17.722745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:17.722759 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:17.736476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:17.736437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:17.906057 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:17.906021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:18:26.046818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:26.046754 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:26.869488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:26.869453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:28.965559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:28.965515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:31.462817 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:31.462762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:33.466047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:33.466011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:33.635638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:33.635610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:33.801014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:33.800932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:34.505966 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:34.505933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:34.675576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:34.675536 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:18:34.683122 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:34.683095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:35.321088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:35.321055 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:36.294213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:36.294175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:39.548430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:39.548393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:39.671368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:39.671335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:43.507623 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:43.507580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:43.697326 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:43.697288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:44.191379 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:44.191317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:50.743388 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:50.743350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:50.845056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:50.845024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:50.984695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:50.984661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:51.373115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:51.373078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:51.629771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:51.629679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:18:57.639067 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:57.639025 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:18:59.522514 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:18:59.522471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:00.008399 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:00.008363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:00.107461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:00.107418 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:01.225054 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:01.225020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:02.088310 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:02.088269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:02.880515 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:02.880479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:03.981696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:03.981662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:04.767565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.767527 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:04.771848 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.771822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.777707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.777678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6ad61446-serving-cert\"" Sep 18 22:19:04.787076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.787037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:04.811575 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.811548 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:04.873237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.873194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.873461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.873252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7j87\" (UniqueName: \"kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.873461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.873354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.873461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.873387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.912007 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.911966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:04.974338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.974278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7j87\" (UniqueName: \"kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.974537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.974356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.974537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.974384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.974708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.974682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.975180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.975148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.977264 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.977240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.977393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.977277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:04.983837 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:04.983801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7j87\" (UniqueName: \"kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87\") pod \"maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:05.011854 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.011781 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:05.085063 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.084960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:05.351077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.350971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:05.383516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.383301 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:05.387244 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:19:05.387210 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3c92bd_4ec8_4058_accd_4c49ef58d345.slice/crio-c2bcb997c2c26b8fcfcc35d69851bba8a6106bd49042b5756e2bc597e74b809f WatchSource:0}: Error finding container c2bcb997c2c26b8fcfcc35d69851bba8a6106bd49042b5756e2bc597e74b809f: Status 404 returned error can't find the container with id c2bcb997c2c26b8fcfcc35d69851bba8a6106bd49042b5756e2bc597e74b809f Sep 18 22:19:05.388679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.388658 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:19:05.453519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:05.453457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:06.147348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:06.147254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" event={"ID":"fb3c92bd-4ec8-4058-accd-4c49ef58d345","Type":"ContainerStarted","Data":"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8"} Sep 18 22:19:06.147348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:06.147303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" event={"ID":"fb3c92bd-4ec8-4058-accd-4c49ef58d345","Type":"ContainerStarted","Data":"c2bcb997c2c26b8fcfcc35d69851bba8a6106bd49042b5756e2bc597e74b809f"} Sep 18 22:19:06.147776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:06.147399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:06.170319 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:06.170261 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" podStartSLOduration=1.798113523 podStartE2EDuration="2.170239909s" totalImagesPullingTime="372.126386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:19:06.167332374 +0000 UTC m=+1131.838830341" watchObservedRunningTime="2026-09-18 22:19:06.170239909 +0000 UTC m=+1131.841737880" Sep 18 22:19:09.837019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:09.836983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:10.986745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:10.986708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.084865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.084826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.184731 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.184697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.294408 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.294323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.397502 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.397467 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.728296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.728263 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:11.829306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:11.829269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:12.159041 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:12.158967 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:12.650019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:12.649988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:13.535511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:13.535478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:14.679742 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:14.679698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:15.700654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:15.700617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:16.621393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:16.621362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:17.757813 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:17.757765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:18.828114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:18.828083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:19.554010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:19.553976 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:19.788733 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:19.788697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:19.893725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:19.893622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.131819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.131758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.262377 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.262343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.512235 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.512203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.617830 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.617711 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.713758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.713720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:20.993150 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:20.993106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:21.142609 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:21.142575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:22.447914 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:22.447875 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:22.659275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:22.659237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:22.793863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:22.793731 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:23.909148 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:23.909111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:25.742343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:25.742308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:25.842735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:25.842702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:26.880011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:26.879969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:28.268378 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:28.268347 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:28.372359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:28.372326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:29.017107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:29.017071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:29.058645 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:29.058601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:19:30.017603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:30.017571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:30.996428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:30.996394 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:31.096963 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:31.096930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:31.930919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:31.930885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:32.994487 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:32.994451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:34.032334 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:34.032301 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:38.305947 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.305864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:38.410519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.410480 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:19:38.414430 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.414409 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.418004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.417974 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-268aff\"/\"default-dockercfg-xr88w\"" Sep 18 22:19:38.418322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.418301 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-268aff\"/\"openshift-service-ca.crt\"" Sep 18 22:19:38.418730 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.418700 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-268aff\"/\"kube-root-ca.crt\"" Sep 18 22:19:38.418858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.418841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-268aff\"/\"noten-model-268aff-kserve-self-signed-certs\"" Sep 18 22:19:38.429560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.429524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:19:38.453471 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.453440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:38.508277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.508467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wzm\" (UniqueName: \"kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.508467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.508467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.508467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.508664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.508513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.609812 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.609812 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.609812 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wzm\" (UniqueName: \"kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.609947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.610166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.610220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.610256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.610244 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.612476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.612451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.612851 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.612830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.624457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.624427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wzm\" (UniqueName: \"kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm\") pod \"noten-model-268aff-kserve-6bf5d455bc-mwsgp\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:38.724441 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:38.724406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:39.030525 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:39.030489 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:19:39.032554 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:19:39.032523 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b37b38_d21e_4603_bd75_9b0dda816612.slice/crio-10e97134a46143f9378974a82df0c56a0aa79d4a5b2e9582e7915aee06163d29 WatchSource:0}: Error finding container 10e97134a46143f9378974a82df0c56a0aa79d4a5b2e9582e7915aee06163d29: Status 404 returned error can't find the container with id 10e97134a46143f9378974a82df0c56a0aa79d4a5b2e9582e7915aee06163d29 Sep 18 22:19:39.333676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:39.333577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" event={"ID":"88b37b38-d21e-4603-bd75-9b0dda816612","Type":"ContainerStarted","Data":"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1"} Sep 18 22:19:39.333676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:39.333629 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" event={"ID":"88b37b38-d21e-4603-bd75-9b0dda816612","Type":"ContainerStarted","Data":"10e97134a46143f9378974a82df0c56a0aa79d4a5b2e9582e7915aee06163d29"} Sep 18 22:19:39.358864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:39.358807 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" podStartSLOduration=1.3587696949999999 podStartE2EDuration="1.358769695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:19:39.358201972 +0000 UTC m=+1165.029699957" watchObservedRunningTime="2026-09-18 22:19:39.358769695 +0000 UTC m=+1165.030267661" Sep 18 22:19:42.523231 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.523196 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:42.523726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.523415 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" podUID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" containerName="maas-api" containerID="cri-o://c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8" gracePeriod=30 Sep 18 22:19:42.563491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.563458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:42.696560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.696525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:42.793348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.793319 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:42.868233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.868198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:42.951522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.951486 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs\") pod \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " Sep 18 22:19:42.951522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.951533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls\") pod \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " Sep 18 22:19:42.951762 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.951568 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7j87\" (UniqueName: \"kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87\") pod \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " Sep 18 22:19:42.951762 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.951596 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca\") pod \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\" (UID: \"fb3c92bd-4ec8-4058-accd-4c49ef58d345\") " Sep 18 22:19:42.952151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.952117 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb3c92bd-4ec8-4058-accd-4c49ef58d345" (UID: "fb3c92bd-4ec8-4058-accd-4c49ef58d345"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:19:42.954087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.954051 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87" (OuterVolumeSpecName: "kube-api-access-d7j87") pod "fb3c92bd-4ec8-4058-accd-4c49ef58d345" (UID: "fb3c92bd-4ec8-4058-accd-4c49ef58d345"). InnerVolumeSpecName "kube-api-access-d7j87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:19:42.954641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.954607 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb3c92bd-4ec8-4058-accd-4c49ef58d345" (UID: "fb3c92bd-4ec8-4058-accd-4c49ef58d345"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:19:42.954641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.954622 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb3c92bd-4ec8-4058-accd-4c49ef58d345" (UID: "fb3c92bd-4ec8-4058-accd-4c49ef58d345"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:19:42.995617 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:42.995574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:43.052658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.052564 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:19:43.052658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.052594 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3c92bd-4ec8-4058-accd-4c49ef58d345-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:19:43.052658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.052605 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7j87\" (UniqueName: \"kubernetes.io/projected/fb3c92bd-4ec8-4058-accd-4c49ef58d345-kube-api-access-d7j87\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:19:43.052658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.052613 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3c92bd-4ec8-4058-accd-4c49ef58d345-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:19:43.102046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.102017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:43.355638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.355541 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" containerID="c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8" exitCode=0 Sep 18 22:19:43.355638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.355597 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" Sep 18 22:19:43.355638 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.355621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" event={"ID":"fb3c92bd-4ec8-4058-accd-4c49ef58d345","Type":"ContainerDied","Data":"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8"} Sep 18 22:19:43.356088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.355674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc" event={"ID":"fb3c92bd-4ec8-4058-accd-4c49ef58d345","Type":"ContainerDied","Data":"c2bcb997c2c26b8fcfcc35d69851bba8a6106bd49042b5756e2bc597e74b809f"} Sep 18 22:19:43.356088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.355696 2577 scope.go:117] "RemoveContainer" containerID="c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8" Sep 18 22:19:43.365895 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.365868 2577 scope.go:117] "RemoveContainer" containerID="c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8" Sep 18 22:19:43.366294 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:19:43.366268 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8\": container with ID starting with c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8 not found: ID does not exist" containerID="c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8" Sep 18 22:19:43.366357 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.366305 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8"} err="failed to get container status \"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8\": rpc error: code = NotFound desc = could not find container \"c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8\": container with ID starting with c89026923efc5366eb351a68ebf639778ac3e5543e41280da84fe7d130656ef8 not found: ID does not exist" Sep 18 22:19:43.385255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.385222 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:43.389643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.389610 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad61446-7fd5df4db5-fs5zc"] Sep 18 22:19:43.410674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.410613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:43.534846 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:43.534818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:44.632025 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:44.631980 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:44.849261 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:44.849227 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" path="/var/lib/kubelet/pods/fb3c92bd-4ec8-4058-accd-4c49ef58d345/volumes" Sep 18 22:19:45.663429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:45.663387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:47.615242 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:47.615208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:48.725361 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:48.725325 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:48.725832 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:48.725378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:48.738377 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:48.738347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:49.401336 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:49.401304 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:19:50.749028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:50.748984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:19:51.082343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.082252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:51.303563 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.303529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:51.317651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.317614 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:19:51.507102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.507071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:51.650324 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.650298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:51.964654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:51.964619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:52.210105 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:52.210048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:53.713994 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:53.713964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:19:56.913489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:19:56.913454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:00.224606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:00.224575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:00.370868 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:00.370829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:00.504005 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:00.503908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:00.638127 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:00.638093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:06.422726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.422664 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" podUID="88b37b38-d21e-4603-bd75-9b0dda816612" containerName="main" containerID="cri-o://d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1" gracePeriod=45 Sep 18 22:20:06.676239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.676163 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:20:06.790233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7wzm\" (UniqueName: \"kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790241 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790310 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790404 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790425 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790448 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm\") pod \"88b37b38-d21e-4603-bd75-9b0dda816612\" (UID: \"88b37b38-d21e-4603-bd75-9b0dda816612\") " Sep 18 22:20:06.790712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home" (OuterVolumeSpecName: "home") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:06.790712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache" (OuterVolumeSpecName: "model-cache") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:06.790712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790675 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:06.790898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790879 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:06.790941 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790902 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:06.790941 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.790910 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:06.792877 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.792846 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm" (OuterVolumeSpecName: "dshm") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:06.793138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.793116 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:20:06.793288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.793267 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm" (OuterVolumeSpecName: "kube-api-access-l7wzm") pod "88b37b38-d21e-4603-bd75-9b0dda816612" (UID: "88b37b38-d21e-4603-bd75-9b0dda816612"). InnerVolumeSpecName "kube-api-access-l7wzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:20:06.891937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.891897 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wzm\" (UniqueName: \"kubernetes.io/projected/88b37b38-d21e-4603-bd75-9b0dda816612-kube-api-access-l7wzm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:06.891937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.891930 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b37b38-d21e-4603-bd75-9b0dda816612-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:06.891937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:06.891939 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b37b38-d21e-4603-bd75-9b0dda816612-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:07.487783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.487740 2577 generic.go:356] "Generic (PLEG): container finished" podID="88b37b38-d21e-4603-bd75-9b0dda816612" containerID="d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1" exitCode=0 Sep 18 22:20:07.488227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.487822 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" Sep 18 22:20:07.488227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.487833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" event={"ID":"88b37b38-d21e-4603-bd75-9b0dda816612","Type":"ContainerDied","Data":"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1"} Sep 18 22:20:07.488227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.487894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp" event={"ID":"88b37b38-d21e-4603-bd75-9b0dda816612","Type":"ContainerDied","Data":"10e97134a46143f9378974a82df0c56a0aa79d4a5b2e9582e7915aee06163d29"} Sep 18 22:20:07.488227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.487912 2577 scope.go:117] "RemoveContainer" containerID="d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1" Sep 18 22:20:07.497093 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.497071 2577 scope.go:117] "RemoveContainer" containerID="d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1" Sep 18 22:20:07.497497 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:20:07.497473 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1\": container with ID starting with d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1 not found: ID does not exist" containerID="d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1" Sep 18 22:20:07.497585 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.497504 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1"} err="failed to get container status \"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1\": rpc error: code = NotFound desc = could not find container \"d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1\": container with ID starting with d8f74ce5016f4abf5143487c7964a7aa25737e76d23dcafe9aef1d547f4802c1 not found: ID does not exist" Sep 18 22:20:07.527627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.527590 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:20:07.531333 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:07.531299 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-268aff/noten-model-268aff-kserve-6bf5d455bc-mwsgp"] Sep 18 22:20:08.717872 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:08.717843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:08.849261 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:08.849222 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b37b38-d21e-4603-bd75-9b0dda816612" path="/var/lib/kubelet/pods/88b37b38-d21e-4603-bd75-9b0dda816612/volumes" Sep 18 22:20:09.116353 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:09.116267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:09.323071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:09.323045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:09.992107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:09.992072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:10.453230 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:10.453195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:10.628349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:10.628320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:11.419629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:11.419596 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:20:11.487546 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:11.487507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:12.875532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:12.875505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:13.475632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:13.475602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:13.623743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:13.623709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:14.399575 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:14.399543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:15.240972 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:15.240931 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:20:15.941344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:15.941303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:17.414598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:17.414565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:18.661700 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:18.661663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:18.765538 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:18.765503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:19.229648 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:19.229612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:19.324584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:19.324550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:26.439998 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.439923 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" podUID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" containerName="main" containerID="cri-o://61d9bbd0f28a040d4aa6172640c4ac740267e31f38e5e737db554344fcb92307" gracePeriod=45 Sep 18 22:20:26.594123 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.594088 2577 generic.go:356] "Generic (PLEG): container finished" podID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" containerID="61d9bbd0f28a040d4aa6172640c4ac740267e31f38e5e737db554344fcb92307" exitCode=0 Sep 18 22:20:26.594301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.594165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" event={"ID":"114773e5-5ef2-4f1b-9ccb-f55e204fed49","Type":"ContainerDied","Data":"61d9bbd0f28a040d4aa6172640c4ac740267e31f38e5e737db554344fcb92307"} Sep 18 22:20:26.701621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.701545 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:20:26.798233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798199 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp9jt\" (UniqueName: \"kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798238 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798311 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798338 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir\") pod \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\" (UID: \"114773e5-5ef2-4f1b-9ccb-f55e204fed49\") " Sep 18 22:20:26.798708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798555 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache" (OuterVolumeSpecName: "model-cache") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:26.798708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798565 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home" (OuterVolumeSpecName: "home") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:26.798708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.798661 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:26.800889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.800852 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm" (OuterVolumeSpecName: "dshm") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:20:26.800889 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.800868 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt" (OuterVolumeSpecName: "kube-api-access-cp9jt") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "kube-api-access-cp9jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:20:26.801051 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.800942 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "114773e5-5ef2-4f1b-9ccb-f55e204fed49" (UID: "114773e5-5ef2-4f1b-9ccb-f55e204fed49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:20:26.899409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899378 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:26.899409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899407 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:26.899409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899418 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:26.899657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899427 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9jt\" (UniqueName: \"kubernetes.io/projected/114773e5-5ef2-4f1b-9ccb-f55e204fed49-kube-api-access-cp9jt\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:26.899657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899435 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:26.899657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:26.899443 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114773e5-5ef2-4f1b-9ccb-f55e204fed49-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:27.600921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:27.600892 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" Sep 18 22:20:27.601337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:27.600890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs" event={"ID":"114773e5-5ef2-4f1b-9ccb-f55e204fed49","Type":"ContainerDied","Data":"f8cdfd77ef954a2887b42f2f3a920e9a8b3bafdcf56e6a031e88a76c4062d675"} Sep 18 22:20:27.601337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:27.601018 2577 scope.go:117] "RemoveContainer" containerID="61d9bbd0f28a040d4aa6172640c4ac740267e31f38e5e737db554344fcb92307" Sep 18 22:20:27.627295 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:27.627261 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:20:27.629715 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:27.629685 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3d28c1/test-model-3d28c1-kserve-556db7c499-dbpzs"] Sep 18 22:20:28.849609 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:28.849572 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" path="/var/lib/kubelet/pods/114773e5-5ef2-4f1b-9ccb-f55e204fed49/volumes" Sep 18 22:20:33.474121 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:33.474081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:33.844639 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:33.844555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:34.829420 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:34.829384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:35.790359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:35.790326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:36.435967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:36.435922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:36.582350 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:36.582319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:37.622511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:37.622476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:37.987767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:37.987726 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:39.316045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:39.316017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:40.431256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:40.431223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:40.849936 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:40.849905 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:20:41.899974 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:41.899945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:43.026808 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:43.026765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:43.518302 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:43.518271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:43.816780 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:43.816694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:44.191210 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:44.191182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:44.337373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:44.337336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:45.004337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:45.004308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:45.771141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:45.771107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:46.909866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:46.909834 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:20:46.910259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:46.910031 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" podUID="ac59a79b-812f-4b62-b0f8-18e4e2132866" containerName="maas-api" containerID="cri-o://8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f" gracePeriod=30 Sep 18 22:20:47.043727 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.043683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:47.175901 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.175827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:20:47.312813 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.312091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") pod \"ac59a79b-812f-4b62-b0f8-18e4e2132866\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " Sep 18 22:20:47.312813 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.312153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca\") pod \"ac59a79b-812f-4b62-b0f8-18e4e2132866\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " Sep 18 22:20:47.312813 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.312212 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqwbl\" (UniqueName: \"kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl\") pod \"ac59a79b-812f-4b62-b0f8-18e4e2132866\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " Sep 18 22:20:47.312813 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.312265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs\") pod \"ac59a79b-812f-4b62-b0f8-18e4e2132866\" (UID: \"ac59a79b-812f-4b62-b0f8-18e4e2132866\") " Sep 18 22:20:47.313504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.313473 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac59a79b-812f-4b62-b0f8-18e4e2132866" (UID: "ac59a79b-812f-4b62-b0f8-18e4e2132866"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:20:47.326359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.326323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl" (OuterVolumeSpecName: "kube-api-access-kqwbl") pod "ac59a79b-812f-4b62-b0f8-18e4e2132866" (UID: "ac59a79b-812f-4b62-b0f8-18e4e2132866"). InnerVolumeSpecName "kube-api-access-kqwbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:20:47.327008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.326980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac59a79b-812f-4b62-b0f8-18e4e2132866" (UID: "ac59a79b-812f-4b62-b0f8-18e4e2132866"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:20:47.334348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.334191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:47.338959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.338908 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac59a79b-812f-4b62-b0f8-18e4e2132866" (UID: "ac59a79b-812f-4b62-b0f8-18e4e2132866"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:20:47.413338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.413300 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:47.413338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.413332 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac59a79b-812f-4b62-b0f8-18e4e2132866-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:47.413338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.413342 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwbl\" (UniqueName: \"kubernetes.io/projected/ac59a79b-812f-4b62-b0f8-18e4e2132866-kube-api-access-kqwbl\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:47.413338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.413350 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac59a79b-812f-4b62-b0f8-18e4e2132866-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:20:47.617844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.617814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:47.710529 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.710497 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac59a79b-812f-4b62-b0f8-18e4e2132866" containerID="8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f" exitCode=0 Sep 18 22:20:47.710712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.710557 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" Sep 18 22:20:47.710712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.710586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" event={"ID":"ac59a79b-812f-4b62-b0f8-18e4e2132866","Type":"ContainerDied","Data":"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f"} Sep 18 22:20:47.710712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.710649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr" event={"ID":"ac59a79b-812f-4b62-b0f8-18e4e2132866","Type":"ContainerDied","Data":"77e7159838ca124f534aea58da9723cf863fbdb24398661074cff80ac0068e51"} Sep 18 22:20:47.710712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.710673 2577 scope.go:117] "RemoveContainer" containerID="8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f" Sep 18 22:20:47.731923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.731892 2577 scope.go:117] "RemoveContainer" containerID="8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f" Sep 18 22:20:47.732423 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:20:47.732386 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f\": container with ID starting with 8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f not found: ID does not exist" containerID="8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f" Sep 18 22:20:47.732534 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.732434 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f"} err="failed to get container status \"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f\": rpc error: code = NotFound desc = could not find container \"8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f\": container with ID starting with 8347dd6d619d08165f8a0915d54db7fc1077edd3251bf19a2c7e63131d81975f not found: ID does not exist" Sep 18 22:20:47.751044 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.751011 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:20:47.755779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.755747 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d28c1-5675f58b88-bxtgr"] Sep 18 22:20:47.775276 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:47.775244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:48.104647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:48.104618 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:48.371598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:48.371515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:48.695092 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:48.695066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:48.848200 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:48.848162 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac59a79b-812f-4b62-b0f8-18e4e2132866" path="/var/lib/kubelet/pods/ac59a79b-812f-4b62-b0f8-18e4e2132866/volumes" Sep 18 22:20:49.800603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:49.800559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:51.406038 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:51.406002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:52.922198 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:52.922148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:54.596764 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:54.596720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:54.763518 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:54.763485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:55.312368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:55.312324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:55.778526 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:55.778497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:56.041362 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:56.041261 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:56.789500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:56.789466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:57.113267 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:57.113167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:58.467341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:58.467306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:58.629953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:58.629920 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:59.783332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:59.783299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:20:59.946541 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:20:59.946506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:00.252907 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:00.252852 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:00.872763 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:00.872724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:01.568142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:01.568097 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:21:03.637131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:03.637091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:04.404408 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:04.404378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:04.596948 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:04.596918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:04.997721 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:04.997676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:06.354025 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:06.353949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:07.747908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:07.747872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:07.976232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:07.976190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:08.596245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596202 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-6c58a3"] Sep 18 22:21:08.596704 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596683 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" containerName="maas-api" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596706 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" containerName="maas-api" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596716 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac59a79b-812f-4b62-b0f8-18e4e2132866" containerName="maas-api" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596723 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac59a79b-812f-4b62-b0f8-18e4e2132866" containerName="maas-api" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596742 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b37b38-d21e-4603-bd75-9b0dda816612" containerName="main" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596751 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b37b38-d21e-4603-bd75-9b0dda816612" containerName="main" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596765 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" containerName="main" Sep 18 22:21:08.596818 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596773 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" containerName="main" Sep 18 22:21:08.597175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596921 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac59a79b-812f-4b62-b0f8-18e4e2132866" containerName="maas-api" Sep 18 22:21:08.597175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596936 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3c92bd-4ec8-4058-accd-4c49ef58d345" containerName="maas-api" Sep 18 22:21:08.597175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596945 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="114773e5-5ef2-4f1b-9ccb-f55e204fed49" containerName="main" Sep 18 22:21:08.597175 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.596954 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b37b38-d21e-4603-bd75-9b0dda816612" containerName="main" Sep 18 22:21:08.601495 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.601470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:08.607026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.606990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t97qt\"" Sep 18 22:21:08.613378 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.613349 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-6c58a3"] Sep 18 22:21:08.728125 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.728083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69csz\" (UniqueName: \"kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz\") pod \"test-curl-7639-6c58a3\" (UID: \"55d9da4a-6983-43c0-859b-e02fc9b7b156\") " pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:08.829032 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.828985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69csz\" (UniqueName: \"kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz\") pod \"test-curl-7639-6c58a3\" (UID: \"55d9da4a-6983-43c0-859b-e02fc9b7b156\") " pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:08.843344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.843312 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69csz\" (UniqueName: \"kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz\") pod \"test-curl-7639-6c58a3\" (UID: \"55d9da4a-6983-43c0-859b-e02fc9b7b156\") " pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:08.911309 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:08.911201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:09.209740 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:09.209710 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-6c58a3"] Sep 18 22:21:09.212859 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:09.212828 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d9da4a_6983_43c0_859b_e02fc9b7b156.slice/crio-e61f29d8eaa5d4e78fcff20fc054b29b5e1f92e7925adc3f333b114cc61ceced WatchSource:0}: Error finding container e61f29d8eaa5d4e78fcff20fc054b29b5e1f92e7925adc3f333b114cc61ceced: Status 404 returned error can't find the container with id e61f29d8eaa5d4e78fcff20fc054b29b5e1f92e7925adc3f333b114cc61ceced Sep 18 22:21:09.723660 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:09.723630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:09.828476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:09.828439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-6c58a3" event={"ID":"55d9da4a-6983-43c0-859b-e02fc9b7b156","Type":"ContainerStarted","Data":"2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6"} Sep 18 22:21:09.828476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:09.828483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-6c58a3" event={"ID":"55d9da4a-6983-43c0-859b-e02fc9b7b156","Type":"ContainerStarted","Data":"e61f29d8eaa5d4e78fcff20fc054b29b5e1f92e7925adc3f333b114cc61ceced"} Sep 18 22:21:09.902702 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:09.902654 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7639-6c58a3" podStartSLOduration=1.902638015 podStartE2EDuration="1.902638015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:09.901242467 +0000 UTC m=+1255.572740437" watchObservedRunningTime="2026-09-18 22:21:09.902638015 +0000 UTC m=+1255.574135981" Sep 18 22:21:11.307349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:11.307311 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-6c58a3"] Sep 18 22:21:11.317909 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:11.317873 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-6c58a3"] Sep 18 22:21:11.839536 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:11.839495 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7639-6c58a3" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" containerName="test-curl-7639-6c58a3" containerID="cri-o://2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6" gracePeriod=2 Sep 18 22:21:11.844085 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:11.844051 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-6c58a3\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:12.101158 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:12.101039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:13.521228 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.521195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:13.762026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.761990 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-393fc3"] Sep 18 22:21:13.762380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.762367 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" containerName="test-curl-7639-6c58a3" Sep 18 22:21:13.762432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.762381 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" containerName="test-curl-7639-6c58a3" Sep 18 22:21:13.762499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.762490 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" containerName="test-curl-7639-6c58a3" Sep 18 22:21:13.765504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.765484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:13.774017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.773753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jqd\" (UniqueName: \"kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd\") pod \"test-curl-7639-393fc3\" (UID: \"cc7be18e-a232-4fb3-82f2-36972ef5eda0\") " pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:13.775577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.775552 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-393fc3"] Sep 18 22:21:13.875236 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.875195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jqd\" (UniqueName: \"kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd\") pod \"test-curl-7639-393fc3\" (UID: \"cc7be18e-a232-4fb3-82f2-36972ef5eda0\") " pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:13.884457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:13.884422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jqd\" (UniqueName: \"kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd\") pod \"test-curl-7639-393fc3\" (UID: \"cc7be18e-a232-4fb3-82f2-36972ef5eda0\") " pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:14.036069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.036000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:14.075922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.075886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:14.077205 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.077183 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69csz\" (UniqueName: \"kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz\") pod \"55d9da4a-6983-43c0-859b-e02fc9b7b156\" (UID: \"55d9da4a-6983-43c0-859b-e02fc9b7b156\") " Sep 18 22:21:14.079600 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.079566 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz" (OuterVolumeSpecName: "kube-api-access-69csz") pod "55d9da4a-6983-43c0-859b-e02fc9b7b156" (UID: "55d9da4a-6983-43c0-859b-e02fc9b7b156"). InnerVolumeSpecName "kube-api-access-69csz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:21:14.178840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.178803 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69csz\" (UniqueName: \"kubernetes.io/projected/55d9da4a-6983-43c0-859b-e02fc9b7b156-kube-api-access-69csz\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:14.339213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.339130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:14.351533 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.351505 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-393fc3"] Sep 18 22:21:14.353473 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:14.353443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc7be18e_a232_4fb3_82f2_36972ef5eda0.slice/crio-7574b52be0256c511764d1e1b49a1aee08a90eba9408b4c91b99bb959108c940 WatchSource:0}: Error finding container 7574b52be0256c511764d1e1b49a1aee08a90eba9408b4c91b99bb959108c940: Status 404 returned error can't find the container with id 7574b52be0256c511764d1e1b49a1aee08a90eba9408b4c91b99bb959108c940 Sep 18 22:21:14.849848 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.849816 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d9da4a-6983-43c0-859b-e02fc9b7b156" path="/var/lib/kubelet/pods/55d9da4a-6983-43c0-859b-e02fc9b7b156/volumes" Sep 18 22:21:14.856488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.856448 2577 generic.go:356] "Generic (PLEG): container finished" podID="55d9da4a-6983-43c0-859b-e02fc9b7b156" containerID="2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6" exitCode=137 Sep 18 22:21:14.856488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.856496 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-6c58a3" Sep 18 22:21:14.856732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.856536 2577 scope.go:117] "RemoveContainer" containerID="2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6" Sep 18 22:21:14.858645 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.858614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-393fc3" event={"ID":"cc7be18e-a232-4fb3-82f2-36972ef5eda0","Type":"ContainerStarted","Data":"1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110"} Sep 18 22:21:14.858776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.858665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-393fc3" event={"ID":"cc7be18e-a232-4fb3-82f2-36972ef5eda0","Type":"ContainerStarted","Data":"7574b52be0256c511764d1e1b49a1aee08a90eba9408b4c91b99bb959108c940"} Sep 18 22:21:14.867985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.867966 2577 scope.go:117] "RemoveContainer" containerID="2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6" Sep 18 22:21:14.868303 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:14.868282 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6\": container with ID starting with 2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6 not found: ID does not exist" containerID="2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6" Sep 18 22:21:14.868368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.868316 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6"} err="failed to get container status \"2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6\": rpc error: code = NotFound desc = could not find container \"2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6\": container with ID starting with 2027ec319e20835ceb8c8d6adf358830c35d9a5b17a8d315f88c5206ad3a27b6 not found: ID does not exist" Sep 18 22:21:14.885912 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:14.885862 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7639-393fc3" podStartSLOduration=1.8858467110000001 podStartE2EDuration="1.885846711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:14.884495023 +0000 UTC m=+1260.555992994" watchObservedRunningTime="2026-09-18 22:21:14.885846711 +0000 UTC m=+1260.557344680" Sep 18 22:21:15.190052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.190017 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:21:15.190287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.190263 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" podUID="105b066f-c994-4c45-8f14-fee0d2622291" containerName="maas-api" containerID="cri-o://aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7" gracePeriod=30 Sep 18 22:21:15.270114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.270081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:15.424577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.424550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:15.457836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.457734 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:21:15.493144 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.493114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") pod \"105b066f-c994-4c45-8f14-fee0d2622291\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " Sep 18 22:21:15.493340 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.493158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs\") pod \"105b066f-c994-4c45-8f14-fee0d2622291\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " Sep 18 22:21:15.493340 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.493195 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5pkv\" (UniqueName: \"kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv\") pod \"105b066f-c994-4c45-8f14-fee0d2622291\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " Sep 18 22:21:15.493340 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.493268 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca\") pod \"105b066f-c994-4c45-8f14-fee0d2622291\" (UID: \"105b066f-c994-4c45-8f14-fee0d2622291\") " Sep 18 22:21:15.493709 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.493676 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "105b066f-c994-4c45-8f14-fee0d2622291" (UID: "105b066f-c994-4c45-8f14-fee0d2622291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:21:15.495658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.495628 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "105b066f-c994-4c45-8f14-fee0d2622291" (UID: "105b066f-c994-4c45-8f14-fee0d2622291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:21:15.495658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.495649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "105b066f-c994-4c45-8f14-fee0d2622291" (UID: "105b066f-c994-4c45-8f14-fee0d2622291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:21:15.495893 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.495693 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv" (OuterVolumeSpecName: "kube-api-access-b5pkv") pod "105b066f-c994-4c45-8f14-fee0d2622291" (UID: "105b066f-c994-4c45-8f14-fee0d2622291"). InnerVolumeSpecName "kube-api-access-b5pkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:21:15.534989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.534953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:15.594641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.594602 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:15.594641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.594630 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105b066f-c994-4c45-8f14-fee0d2622291-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:15.594641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.594641 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5pkv\" (UniqueName: \"kubernetes.io/projected/105b066f-c994-4c45-8f14-fee0d2622291-kube-api-access-b5pkv\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:15.594641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.594650 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b066f-c994-4c45-8f14-fee0d2622291-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:15.595589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.595544 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-393fc3"] Sep 18 22:21:15.604714 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.604677 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-393fc3"] Sep 18 22:21:15.636149 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.636119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:15.737194 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737107 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-209a9d"] Sep 18 22:21:15.737500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737487 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105b066f-c994-4c45-8f14-fee0d2622291" containerName="maas-api" Sep 18 22:21:15.737552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737501 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="105b066f-c994-4c45-8f14-fee0d2622291" containerName="maas-api" Sep 18 22:21:15.737552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737516 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" containerName="test-curl-7639-393fc3" Sep 18 22:21:15.737552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737523 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" containerName="test-curl-7639-393fc3" Sep 18 22:21:15.737664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737582 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" containerName="test-curl-7639-393fc3" Sep 18 22:21:15.737664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.737594 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="105b066f-c994-4c45-8f14-fee0d2622291" containerName="maas-api" Sep 18 22:21:15.740861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.740837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:15.753949 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.753913 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-209a9d"] Sep 18 22:21:15.763186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.763124 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:15.796984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.796944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b5d\" (UniqueName: \"kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d\") pod \"test-curl-7639-209a9d\" (UID: \"4754a561-196f-4d2b-9560-f11f14ee708c\") " pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:15.864964 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.864927 2577 generic.go:356] "Generic (PLEG): container finished" podID="105b066f-c994-4c45-8f14-fee0d2622291" containerID="aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7" exitCode=0 Sep 18 22:21:15.865359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.864976 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" Sep 18 22:21:15.865359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.865012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" event={"ID":"105b066f-c994-4c45-8f14-fee0d2622291","Type":"ContainerDied","Data":"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7"} Sep 18 22:21:15.865359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.865070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj" event={"ID":"105b066f-c994-4c45-8f14-fee0d2622291","Type":"ContainerDied","Data":"d4ead0d96c46874d3116769d1fa5edaf3e9cf7ad83d340c9b10dddd8eb1ab89b"} Sep 18 22:21:15.865359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.865103 2577 scope.go:117] "RemoveContainer" containerID="aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7" Sep 18 22:21:15.865572 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:15.865397 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7639-393fc3" container="test-curl-7639-393fc3" Sep 18 22:21:15.875371 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.875353 2577 scope.go:117] "RemoveContainer" containerID="aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7" Sep 18 22:21:15.875703 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:15.875680 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7\": container with ID starting with aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7 not found: ID does not exist" containerID="aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7" Sep 18 22:21:15.875805 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.875710 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7"} err="failed to get container status \"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7\": rpc error: code = NotFound desc = could not find container \"aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7\": container with ID starting with aea42bb63dd5e7ea6f4cb3424e3d51cc4781133fb9106065c86284937c0057d7 not found: ID does not exist" Sep 18 22:21:15.892613 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.892574 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:21:15.898437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.898409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b5d\" (UniqueName: \"kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d\") pod \"test-curl-7639-209a9d\" (UID: \"4754a561-196f-4d2b-9560-f11f14ee708c\") " pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:15.898603 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.898584 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab559b-6dbdc49648-wm7wj"] Sep 18 22:21:15.912194 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.912156 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b5d\" (UniqueName: \"kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d\") pod \"test-curl-7639-209a9d\" (UID: \"4754a561-196f-4d2b-9560-f11f14ee708c\") " pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:15.998038 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:15.997951 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:16.050835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.050798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:16.172195 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.172158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:16.337840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.337740 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-209a9d"] Sep 18 22:21:16.340959 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:16.340927 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4754a561_196f_4d2b_9560_f11f14ee708c.slice/crio-8636df1d7dcebbed0283f9ad44c21c9a6ba6f063f61bd45817d62e974e5d26d1 WatchSource:0}: Error finding container 8636df1d7dcebbed0283f9ad44c21c9a6ba6f063f61bd45817d62e974e5d26d1: Status 404 returned error can't find the container with id 8636df1d7dcebbed0283f9ad44c21c9a6ba6f063f61bd45817d62e974e5d26d1 Sep 18 22:21:16.849904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.849871 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105b066f-c994-4c45-8f14-fee0d2622291" path="/var/lib/kubelet/pods/105b066f-c994-4c45-8f14-fee0d2622291/volumes" Sep 18 22:21:16.872059 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.872021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-209a9d" event={"ID":"4754a561-196f-4d2b-9560-f11f14ee708c","Type":"ContainerStarted","Data":"fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd"} Sep 18 22:21:16.872547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.872071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-209a9d" event={"ID":"4754a561-196f-4d2b-9560-f11f14ee708c","Type":"ContainerStarted","Data":"8636df1d7dcebbed0283f9ad44c21c9a6ba6f063f61bd45817d62e974e5d26d1"} Sep 18 22:21:16.873153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.873129 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7639-393fc3" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" containerName="test-curl-7639-393fc3" containerID="cri-o://1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110" gracePeriod=2 Sep 18 22:21:16.924542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:16.924487 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7639-209a9d" podStartSLOduration=1.924469274 podStartE2EDuration="1.924469274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:16.923508328 +0000 UTC m=+1262.595006297" watchObservedRunningTime="2026-09-18 22:21:16.924469274 +0000 UTC m=+1262.595967244" Sep 18 22:21:17.927151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:17.927049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:17.978669 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:17.978624 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-209a9d"] Sep 18 22:21:17.988186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:17.988154 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-209a9d"] Sep 18 22:21:18.685420 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:18.685390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:18.884491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:18.884455 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7639-209a9d" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" containerName="test-curl-7639-209a9d" containerID="cri-o://fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd" gracePeriod=2 Sep 18 22:21:19.037563 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.037320 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:19.129824 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.129776 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78jqd\" (UniqueName: \"kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd\") pod \"cc7be18e-a232-4fb3-82f2-36972ef5eda0\" (UID: \"cc7be18e-a232-4fb3-82f2-36972ef5eda0\") " Sep 18 22:21:19.132138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.132106 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd" (OuterVolumeSpecName: "kube-api-access-78jqd") pod "cc7be18e-a232-4fb3-82f2-36972ef5eda0" (UID: "cc7be18e-a232-4fb3-82f2-36972ef5eda0"). InnerVolumeSpecName "kube-api-access-78jqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:21:19.163676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.163639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:19.231291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.231198 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jqd\" (UniqueName: \"kubernetes.io/projected/cc7be18e-a232-4fb3-82f2-36972ef5eda0-kube-api-access-78jqd\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:19.367662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.367629 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:19.581968 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.581882 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:19.891212 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.891128 2577 generic.go:356] "Generic (PLEG): container finished" podID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" containerID="1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110" exitCode=137 Sep 18 22:21:19.891212 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.891173 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:19.891431 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.891224 2577 scope.go:117] "RemoveContainer" containerID="1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110" Sep 18 22:21:19.900902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.900883 2577 scope.go:117] "RemoveContainer" containerID="1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110" Sep 18 22:21:19.901242 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:19.901222 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110\": container with ID starting with 1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110 not found: ID does not exist" containerID="1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110" Sep 18 22:21:19.901328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:19.901257 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110"} err="failed to get container status \"1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110\": rpc error: code = NotFound desc = could not find container \"1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110\": container with ID starting with 1e8994eba8edb56ba15bc2a0b6323e6fa833bc193fe8fb0813dc191933829110 not found: ID does not exist" Sep 18 22:21:19.901870 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:19.901849 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-393fc3\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" pod="opendatahub/test-curl-7639-393fc3" Sep 18 22:21:20.574510 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:20.574479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:20.848836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:20.848721 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7be18e-a232-4fb3-82f2-36972ef5eda0" path="/var/lib/kubelet/pods/cc7be18e-a232-4fb3-82f2-36972ef5eda0/volumes" Sep 18 22:21:21.066772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.066740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:21.149216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.149121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6b5d\" (UniqueName: \"kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d\") pod \"4754a561-196f-4d2b-9560-f11f14ee708c\" (UID: \"4754a561-196f-4d2b-9560-f11f14ee708c\") " Sep 18 22:21:21.151460 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.151425 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d" (OuterVolumeSpecName: "kube-api-access-h6b5d") pod "4754a561-196f-4d2b-9560-f11f14ee708c" (UID: "4754a561-196f-4d2b-9560-f11f14ee708c"). InnerVolumeSpecName "kube-api-access-h6b5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:21:21.250109 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.250070 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b5d\" (UniqueName: \"kubernetes.io/projected/4754a561-196f-4d2b-9560-f11f14ee708c-kube-api-access-h6b5d\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:21:21.903348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.903309 2577 generic.go:356] "Generic (PLEG): container finished" podID="4754a561-196f-4d2b-9560-f11f14ee708c" containerID="fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd" exitCode=137 Sep 18 22:21:21.903815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.903370 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:21.903815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.903404 2577 scope.go:117] "RemoveContainer" containerID="fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd" Sep 18 22:21:21.912906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.912882 2577 scope.go:117] "RemoveContainer" containerID="fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd" Sep 18 22:21:21.913247 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:21.913231 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd\": container with ID starting with fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd not found: ID does not exist" containerID="fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd" Sep 18 22:21:21.913291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:21.913254 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd"} err="failed to get container status \"fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd\": rpc error: code = NotFound desc = could not find container \"fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd\": container with ID starting with fbdae04ec8490abee45c3b1f07b417d78a1787d9e45162ccadba69147032b5cd not found: ID does not exist" Sep 18 22:21:21.914364 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:21:21.914340 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-209a9d\" is forbidden: User \"system:node:ip-10-0-133-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-172.ec2.internal' and this object" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" pod="opendatahub/test-curl-7639-209a9d" Sep 18 22:21:22.237457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:22.237418 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:22.492025 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:22.491928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:22.618578 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:22.618540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:22.848056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:22.847911 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" path="/var/lib/kubelet/pods/4754a561-196f-4d2b-9560-f11f14ee708c/volumes" Sep 18 22:21:22.871690 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:22.871652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:21:23.914033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:23.913997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:25.214057 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.214016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:25.457278 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.457247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:25.512091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.512004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:21:25.512404 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.512392 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" containerName="test-curl-7639-209a9d" Sep 18 22:21:25.512445 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.512406 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" containerName="test-curl-7639-209a9d" Sep 18 22:21:25.512488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.512467 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4754a561-196f-4d2b-9560-f11f14ee708c" containerName="test-curl-7639-209a9d" Sep 18 22:21:25.517072 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.517052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.520406 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.520387 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"distinct-w4-59b357-kserve-self-signed-certs\"" Sep 18 22:21:25.534875 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.534835 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:21:25.559466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.559435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:25.592279 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.592465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.592465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfcl\" (UniqueName: \"kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.592465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.592465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.592630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.592471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693604 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693834 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693834 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfcl\" (UniqueName: \"kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.693965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.693911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.694874 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.694845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.695087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.695063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.695172 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.695140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.697231 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.697197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.697376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.697356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.709244 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.709206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfcl\" (UniqueName: \"kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl\") pod \"distinct-w4-59b357-kserve-6865dc8886-cfhr9\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.742611 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.742582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:25.827080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.826975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:25.884712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:25.884680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:26.114652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.114627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:21:26.116003 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:26.115964 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda418aa2d_8df3_4977_98f4_60d36c0adb49.slice/crio-670f7ab0fef07cbc3767fa0de58c5f3ae2b7d367e49b29ea0996032d21218f73 WatchSource:0}: Error finding container 670f7ab0fef07cbc3767fa0de58c5f3ae2b7d367e49b29ea0996032d21218f73: Status 404 returned error can't find the container with id 670f7ab0fef07cbc3767fa0de58c5f3ae2b7d367e49b29ea0996032d21218f73 Sep 18 22:21:26.344999 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.344969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:26.478824 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.478778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:26.693255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.693215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:26.933319 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.933271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" event={"ID":"a418aa2d-8df3-4977-98f4-60d36c0adb49","Type":"ContainerStarted","Data":"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3"} Sep 18 22:21:26.933319 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.933324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" event={"ID":"a418aa2d-8df3-4977-98f4-60d36c0adb49","Type":"ContainerStarted","Data":"670f7ab0fef07cbc3767fa0de58c5f3ae2b7d367e49b29ea0996032d21218f73"} Sep 18 22:21:26.959984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:26.959927 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" podStartSLOduration=1.959911046 podStartE2EDuration="1.959911046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:26.958381286 +0000 UTC m=+1272.629879271" watchObservedRunningTime="2026-09-18 22:21:26.959911046 +0000 UTC m=+1272.631409016" Sep 18 22:21:27.262344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.262260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:27.531593 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.531506 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:21:27.535155 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.535131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.539572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.539545 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-027c8100-serving-cert\"" Sep 18 22:21:27.561142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.561106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:21:27.603244 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.603211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:27.613482 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.613447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.613650 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.613502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6g6\" (UniqueName: \"kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.613650 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.613573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.613815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.613720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.711855 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.711819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:27.714414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.714389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.714562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.714431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6g6\" (UniqueName: \"kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.714562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.714466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.714677 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.714597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.715201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.715176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.717154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.717122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.717306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.717286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.726292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.726253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6g6\" (UniqueName: \"kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6\") pod \"maas-api-e2e-mt-027c8100-9fd479ccb-9nb66\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.846370 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.846256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:27.855343 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:27.855315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:28.146101 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:28.146072 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:21:28.148619 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:28.148583 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b3a95c_3ead_4cdf_85e6_e70171fc093f.slice/crio-dd5e443d9721fe2db0604414b59f05b9bacd34f86013e42ae4cc6b8952782bdb WatchSource:0}: Error finding container dd5e443d9721fe2db0604414b59f05b9bacd34f86013e42ae4cc6b8952782bdb: Status 404 returned error can't find the container with id dd5e443d9721fe2db0604414b59f05b9bacd34f86013e42ae4cc6b8952782bdb Sep 18 22:21:28.865533 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:28.865440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:28.945764 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:28.945732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" event={"ID":"51b3a95c-3ead-4cdf-85e6-e70171fc093f","Type":"ContainerStarted","Data":"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945"} Sep 18 22:21:28.945989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:28.945807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" event={"ID":"51b3a95c-3ead-4cdf-85e6-e70171fc093f","Type":"ContainerStarted","Data":"dd5e443d9721fe2db0604414b59f05b9bacd34f86013e42ae4cc6b8952782bdb"} Sep 18 22:21:28.945989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:28.945854 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:29.002973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:29.002923 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" podStartSLOduration=1.615459703 podStartE2EDuration="2.002904328s" totalImagesPullingTime="387.444625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:29.000451366 +0000 UTC m=+1274.671949356" watchObservedRunningTime="2026-09-18 22:21:29.002904328 +0000 UTC m=+1274.674402373" Sep 18 22:21:29.020321 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:29.020287 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:29.833464 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:29.833431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:30.362696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:30.362668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:30.526544 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:30.526502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:31.256847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:31.256783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:32.742752 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:32.742721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:33.787237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:33.787207 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:34.956765 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:34.956737 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:21:34.974772 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:34.974737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:35.827297 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.827258 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:35.827509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.827311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:35.840442 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.840396 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:35.958548 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.958511 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:21:35.962656 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.962628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:35.965475 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.965452 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-123a30-serving-cert\"" Sep 18 22:21:35.990707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.990670 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:21:35.996867 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:35.996839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:21:36.097464 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.097364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:36.104882 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.104844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.105047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.104956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.105130 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.105053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.105189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.105159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.195101 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.195062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:36.206161 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.206128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.206338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.206181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.206338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.206217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.206338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.206273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.207052 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.207026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.208986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.208957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.208986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.208980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.219952 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.219921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq\") pod \"maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.276964 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.276923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:36.295679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.295651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:36.565118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.565047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:21:36.567157 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:21:36.567123 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b670e63_24b4_41d7_b727_069a41d0d83d.slice/crio-743915c2d0afd9589355a53137daa6201e71d26316436a6a7521402c76db1ee3 WatchSource:0}: Error finding container 743915c2d0afd9589355a53137daa6201e71d26316436a6a7521402c76db1ee3: Status 404 returned error can't find the container with id 743915c2d0afd9589355a53137daa6201e71d26316436a6a7521402c76db1ee3 Sep 18 22:21:36.752528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.752488 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:36.870222 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.870132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:36.990871 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:36.990823 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" event={"ID":"4b670e63-24b4-41d7-b727-069a41d0d83d","Type":"ContainerStarted","Data":"743915c2d0afd9589355a53137daa6201e71d26316436a6a7521402c76db1ee3"} Sep 18 22:21:37.050181 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:37.050147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:37.664552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:37.664521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:38.000438 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:38.000395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" event={"ID":"4b670e63-24b4-41d7-b727-069a41d0d83d","Type":"ContainerStarted","Data":"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b"} Sep 18 22:21:38.000883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:38.000537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:38.052542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:38.052488 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" podStartSLOduration=2.6689624309999997 podStartE2EDuration="3.052470108s" totalImagesPullingTime="383.507677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:38.050969824 +0000 UTC m=+1283.722467806" watchObservedRunningTime="2026-09-18 22:21:38.052470108 +0000 UTC m=+1283.723968074" Sep 18 22:21:39.322654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:39.322613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:40.617403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:40.617364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:41.778917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:41.778885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:42.970980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:42.970946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:44.011391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:44.011364 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:21:44.128443 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:44.128409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:45.848099 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:45.848058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:46.956622 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:46.956593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:47.939542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:47.939504 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:21:48.057363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:48.057330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:49.375414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:49.375378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:50.108924 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:50.108891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:50.373745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:50.373655 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:50.564516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:50.564480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:50.933517 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:50.933480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:51.129576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:51.129546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:51.905625 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:51.905590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:53.280185 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:53.280153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:54.450972 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:54.450939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:54.569539 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:54.569503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:55.662096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:55.662065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:56.740151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:56.740119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:57.941661 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:57.941624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:21:58.919381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:21:58.919343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:00.159504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:00.159467 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:01.094071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.094039 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:22:01.097936 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.097914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.100823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.100767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7f10d8-serving-cert\"" Sep 18 22:22:01.113998 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.113967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:22:01.119859 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.119827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:01.148222 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.148183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.148403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.148307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.148403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.148345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdpd\" (UniqueName: \"kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.148403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.148380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.220417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.220380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.251344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.251494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.251537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdpd\" (UniqueName: \"kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.251592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.252456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.255927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:22:01.256079 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-serving-cert: secret "maas-api-e2e-shared-b-w3-7f10d8-serving-cert" not found Sep 18 22:22:01.256265 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:22:01.256150 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls podName:79ab7647-fa08-419a-8d3b-58dadb46d012 nodeName:}" failed. No retries permitted until 2026-09-18 22:22:01.756129038 +0000 UTC m=+1307.427626993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" (UID: "79ab7647-fa08-419a-8d3b-58dadb46d012") : secret "maas-api-e2e-shared-b-w3-7f10d8-serving-cert" not found Sep 18 22:22:01.269493 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.269462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdpd\" (UniqueName: \"kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.443255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.443223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:01.735133 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.735048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:01.756439 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.756404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.759147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.759119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:01.832243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:01.832207 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:02.009689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:02.009610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:02.289991 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:02.289901 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:22:02.293803 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:22:02.293755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ab7647_fa08_419a_8d3b_58dadb46d012.slice/crio-13ce4cab53bdc6ea620cca4d254b0db2c6e407364f110c82da214f2cc7e30f83 WatchSource:0}: Error finding container 13ce4cab53bdc6ea620cca4d254b0db2c6e407364f110c82da214f2cc7e30f83: Status 404 returned error can't find the container with id 13ce4cab53bdc6ea620cca4d254b0db2c6e407364f110c82da214f2cc7e30f83 Sep 18 22:22:03.133635 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:03.133600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" event={"ID":"79ab7647-fa08-419a-8d3b-58dadb46d012","Type":"ContainerStarted","Data":"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927"} Sep 18 22:22:03.133856 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:03.133661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" event={"ID":"79ab7647-fa08-419a-8d3b-58dadb46d012","Type":"ContainerStarted","Data":"13ce4cab53bdc6ea620cca4d254b0db2c6e407364f110c82da214f2cc7e30f83"} Sep 18 22:22:03.133856 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:03.133690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:03.165476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:03.165417 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" podStartSLOduration=1.827559895 podStartE2EDuration="2.165398075s" totalImagesPullingTime="337.83818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:03.163727458 +0000 UTC m=+1308.835225445" watchObservedRunningTime="2026-09-18 22:22:03.165398075 +0000 UTC m=+1308.836896045" Sep 18 22:22:09.143742 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:09.143711 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:22:11.123778 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:11.123740 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:22:23.802574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:23.802532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:24.706243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:24.706201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:25.332900 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:25.332856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:26.215665 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:26.215631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:27.241661 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:27.241624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:28.242630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:28.242596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:29.029109 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:29.029075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:29.819428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:29.819380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:33.346005 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:33.345954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:42.467353 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.467248 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:22:42.471514 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.471484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.476767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.476739 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-027c8100\"/\"kube-root-ca.crt\"" Sep 18 22:22:42.477696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.477673 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-027c8100\"/\"default-dockercfg-mwsnd\"" Sep 18 22:22:42.477858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.477675 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-027c8100\"/\"openshift-service-ca.crt\"" Sep 18 22:22:42.477858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.477735 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-027c8100\"/\"e2e-lifecycle-model-027c8100-kserve-self-signed-certs\"" Sep 18 22:22:42.483316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.483284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:22:42.556618 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.556590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:42.579858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.579818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.579858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.579874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.580114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.579940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.580114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.580040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvqz\" (UniqueName: \"kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.580114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.580103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.580242 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.580121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvqz\" (UniqueName: \"kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681627 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.681933 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681892 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.682033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.681901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.683847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.683817 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.684135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.684118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.692029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.692001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvqz\" (UniqueName: \"kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz\") pod \"e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:42.782457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:42.782350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:43.067509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:43.067435 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:22:43.069991 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:22:43.069965 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17419ee1_7e92_4a0e_8e2f_ff0f1952fa2f.slice/crio-e550d631dfa5d2dca74b7b5b6b0e7b0fa6055f907dce2ab85ebbb0989fd6cb5f WatchSource:0}: Error finding container e550d631dfa5d2dca74b7b5b6b0e7b0fa6055f907dce2ab85ebbb0989fd6cb5f: Status 404 returned error can't find the container with id e550d631dfa5d2dca74b7b5b6b0e7b0fa6055f907dce2ab85ebbb0989fd6cb5f Sep 18 22:22:43.343226 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:43.343125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" event={"ID":"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f","Type":"ContainerStarted","Data":"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013"} Sep 18 22:22:43.343226 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:43.343170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" event={"ID":"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f","Type":"ContainerStarted","Data":"e550d631dfa5d2dca74b7b5b6b0e7b0fa6055f907dce2ab85ebbb0989fd6cb5f"} Sep 18 22:22:43.363488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:43.363430 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" podStartSLOduration=1.363413926 podStartE2EDuration="1.363413926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:43.362072335 +0000 UTC m=+1349.033570306" watchObservedRunningTime="2026-09-18 22:22:43.363413926 +0000 UTC m=+1349.034911896" Sep 18 22:22:48.543235 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:48.543199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:48.636235 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:48.636191 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:22:48.643560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:48.643524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:22:52.783507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:52.783465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:52.783507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:52.783515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:52.796196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:52.796163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:22:53.403736 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:22:53.403704 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:23:02.429891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.429850 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:23:02.434008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.433983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.437735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.437708 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59b357\"/\"distinct-2-w4-59b357-kserve-self-signed-certs\"" Sep 18 22:23:02.446063 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.446031 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:23:02.484355 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.484355 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.484616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.484616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.484616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.484616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.484563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwz2\" (UniqueName: \"kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.485878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.485850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:02.585897 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.585859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.585897 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.585902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.585953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.585995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.586017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586153 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.586072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwz2\" (UniqueName: \"kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.586349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.586480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.586557 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.586480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.588564 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.588539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.588831 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.588811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.595767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.595735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwz2\" (UniqueName: \"kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2\") pod \"distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:02.745905 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:02.745860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:03.023907 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:03.023880 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:23:03.025936 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:23:03.025908 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6a536d_7f41_42dd_80f8_38da538c222f.slice/crio-3929b08220cf05ec19db770869b64b86b402c6b95c0757c0af30e1d19b5b4f2d WatchSource:0}: Error finding container 3929b08220cf05ec19db770869b64b86b402c6b95c0757c0af30e1d19b5b4f2d: Status 404 returned error can't find the container with id 3929b08220cf05ec19db770869b64b86b402c6b95c0757c0af30e1d19b5b4f2d Sep 18 22:23:03.445179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:03.445145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" event={"ID":"2b6a536d-7f41-42dd-80f8-38da538c222f","Type":"ContainerStarted","Data":"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912"} Sep 18 22:23:03.445179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:03.445191 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" event={"ID":"2b6a536d-7f41-42dd-80f8-38da538c222f","Type":"ContainerStarted","Data":"3929b08220cf05ec19db770869b64b86b402c6b95c0757c0af30e1d19b5b4f2d"} Sep 18 22:23:03.478126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:03.478066 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" podStartSLOduration=1.478049397 podStartE2EDuration="1.478049397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:03.476034178 +0000 UTC m=+1369.147532147" watchObservedRunningTime="2026-09-18 22:23:03.478049397 +0000 UTC m=+1369.149547366" Sep 18 22:23:09.823677 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:09.823637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:23:12.746474 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:12.746437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:12.747001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:12.746498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:12.760517 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:12.760481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:13.512473 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:13.512441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:23:16.021797 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.021751 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:23:16.025542 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.025519 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.029419 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.029388 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"kube-root-ca.crt\"" Sep 18 22:23:16.029584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.029456 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"openshift-service-ca.crt\"" Sep 18 22:23:16.029584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.029388 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"default-dockercfg-jn7hd\"" Sep 18 22:23:16.029584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.029396 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"auth-test-model-123a30-kserve-self-signed-certs\"" Sep 18 22:23:16.037751 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.037709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:23:16.073124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.073093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:16.129715 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.129918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxttb\" (UniqueName: \"kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.129918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.129918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.130031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.130031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.129947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxttb\" (UniqueName: \"kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.231940 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.232000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.231930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.234222 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.234203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.234707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.234690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.250547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.250506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxttb\" (UniqueName: \"kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb\") pod \"auth-test-model-123a30-kserve-59db7f8495-k7hw5\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.338634 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.338535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:16.634365 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:16.634337 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:23:16.635556 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:23:16.635487 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fa44ea7_ddd9_40f8_aac2_a360fb7b077e.slice/crio-0d5588d3377ed54397f288310043c33ce2765ca1940dcbc787e73f6d8765ecdd WatchSource:0}: Error finding container 0d5588d3377ed54397f288310043c33ce2765ca1940dcbc787e73f6d8765ecdd: Status 404 returned error can't find the container with id 0d5588d3377ed54397f288310043c33ce2765ca1940dcbc787e73f6d8765ecdd Sep 18 22:23:17.523586 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:17.523545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" event={"ID":"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e","Type":"ContainerStarted","Data":"d4ae42b480dc0185d69eacd2b07c1422fcf5b3f68bb7fc4350b84994c8782ce6"} Sep 18 22:23:17.523586 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:17.523591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" event={"ID":"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e","Type":"ContainerStarted","Data":"0d5588d3377ed54397f288310043c33ce2765ca1940dcbc787e73f6d8765ecdd"} Sep 18 22:23:17.546879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:17.546825 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" podStartSLOduration=1.5468095800000001 podStartE2EDuration="1.54680958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:17.545088057 +0000 UTC m=+1383.216586027" watchObservedRunningTime="2026-09-18 22:23:17.54680958 +0000 UTC m=+1383.218307541" Sep 18 22:23:26.339176 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:26.339132 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:26.339624 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:26.339196 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:26.352678 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:26.352645 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:26.586047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:26.586015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:23:37.429483 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:37.429430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:23:44.528730 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:44.528676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:45.615992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:45.615943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:46.554826 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:46.554778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:47.471629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:47.471590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:48.442361 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:48.442326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:49.713562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:49.713523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:50.928835 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:50.928772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:51.841397 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:51.841359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:23:56.957632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:23:56.957594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:24:00.046047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.046007 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:24:00.046559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.046247 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" podUID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" containerName="maas-api" containerID="cri-o://b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945" gracePeriod=30 Sep 18 22:24:00.219836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.219772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:00.315714 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.315675 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:24:00.409493 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.409461 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca\") pod \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " Sep 18 22:24:00.409493 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.409502 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs\") pod \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " Sep 18 22:24:00.409770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.409538 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls\") pod \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " Sep 18 22:24:00.409770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.409588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x6g6\" (UniqueName: \"kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6\") pod \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\" (UID: \"51b3a95c-3ead-4cdf-85e6-e70171fc093f\") " Sep 18 22:24:00.409971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.409943 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51b3a95c-3ead-4cdf-85e6-e70171fc093f" (UID: "51b3a95c-3ead-4cdf-85e6-e70171fc093f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:24:00.411988 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.411943 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51b3a95c-3ead-4cdf-85e6-e70171fc093f" (UID: "51b3a95c-3ead-4cdf-85e6-e70171fc093f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:24:00.412110 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.412004 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51b3a95c-3ead-4cdf-85e6-e70171fc093f" (UID: "51b3a95c-3ead-4cdf-85e6-e70171fc093f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:24:00.412110 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.412022 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6" (OuterVolumeSpecName: "kube-api-access-8x6g6") pod "51b3a95c-3ead-4cdf-85e6-e70171fc093f" (UID: "51b3a95c-3ead-4cdf-85e6-e70171fc093f"). InnerVolumeSpecName "kube-api-access-8x6g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:24:00.496495 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.496456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:00.510533 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.510497 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x6g6\" (UniqueName: \"kubernetes.io/projected/51b3a95c-3ead-4cdf-85e6-e70171fc093f-kube-api-access-8x6g6\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:00.510533 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.510532 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b3a95c-3ead-4cdf-85e6-e70171fc093f-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:00.510739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.510547 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:00.510739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.510561 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b3a95c-3ead-4cdf-85e6-e70171fc093f-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:00.728299 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.728263 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:00.764032 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.763997 2577 generic.go:356] "Generic (PLEG): container finished" podID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" containerID="b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945" exitCode=0 Sep 18 22:24:00.764225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.764046 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" Sep 18 22:24:00.764225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.764087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" event={"ID":"51b3a95c-3ead-4cdf-85e6-e70171fc093f","Type":"ContainerDied","Data":"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945"} Sep 18 22:24:00.764225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.764154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66" event={"ID":"51b3a95c-3ead-4cdf-85e6-e70171fc093f","Type":"ContainerDied","Data":"dd5e443d9721fe2db0604414b59f05b9bacd34f86013e42ae4cc6b8952782bdb"} Sep 18 22:24:00.764225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.764182 2577 scope.go:117] "RemoveContainer" containerID="b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945" Sep 18 22:24:00.774730 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.774708 2577 scope.go:117] "RemoveContainer" containerID="b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945" Sep 18 22:24:00.775112 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:24:00.775090 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945\": container with ID starting with b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945 not found: ID does not exist" containerID="b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945" Sep 18 22:24:00.775212 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.775120 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945"} err="failed to get container status \"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945\": rpc error: code = NotFound desc = could not find container \"b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945\": container with ID starting with b464b4a9cf45698077e9657a74e0d99af00f2d3c3ff5ea7ac1cc205e3c9da945 not found: ID does not exist" Sep 18 22:24:00.789518 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.789483 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:24:00.795953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.795918 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-027c8100-9fd479ccb-9nb66"] Sep 18 22:24:00.820535 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.820500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:00.849904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:00.849867 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" path="/var/lib/kubelet/pods/51b3a95c-3ead-4cdf-85e6-e70171fc093f/volumes" Sep 18 22:24:01.314939 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:01.314906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:02.525902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:02.525871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:03.624359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:03.624319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:04.005550 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:04.005516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:04.552507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:04.552469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:05.630391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:05.630352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:06.797082 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:06.797002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:08.978045 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:08.978008 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:24:09.075504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:09.075454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:20.532919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:20.532886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:23.998978 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:23.998921 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" podUID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" containerName="main" containerID="cri-o://42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013" gracePeriod=45 Sep 18 22:24:24.270285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.270252 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:24:24.357050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357220 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357072 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357220 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357106 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357220 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbvqz\" (UniqueName: \"kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache\") pod \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\" (UID: \"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f\") " Sep 18 22:24:24.357381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357322 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:24:24.357550 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357380 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home" (OuterVolumeSpecName: "home") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:24:24.357640 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357590 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:24:24.357696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357650 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.357696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.357671 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.359610 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.359578 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz" (OuterVolumeSpecName: "kube-api-access-gbvqz") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "kube-api-access-gbvqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:24:24.359610 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.359596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm" (OuterVolumeSpecName: "dshm") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:24:24.359761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.359685 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" (UID: "17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:24:24.458486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.458442 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvqz\" (UniqueName: \"kubernetes.io/projected/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-kube-api-access-gbvqz\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.458486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.458478 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.458486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.458489 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.458486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.458496 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:24:24.895509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.895474 2577 generic.go:356] "Generic (PLEG): container finished" podID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" containerID="42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013" exitCode=0 Sep 18 22:24:24.895693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.895518 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" Sep 18 22:24:24.895693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.895553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" event={"ID":"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f","Type":"ContainerDied","Data":"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013"} Sep 18 22:24:24.895693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.895600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt" event={"ID":"17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f","Type":"ContainerDied","Data":"e550d631dfa5d2dca74b7b5b6b0e7b0fa6055f907dce2ab85ebbb0989fd6cb5f"} Sep 18 22:24:24.895693 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.895619 2577 scope.go:117] "RemoveContainer" containerID="42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013" Sep 18 22:24:24.904979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.904954 2577 scope.go:117] "RemoveContainer" containerID="42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013" Sep 18 22:24:24.905330 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:24:24.905311 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013\": container with ID starting with 42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013 not found: ID does not exist" containerID="42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013" Sep 18 22:24:24.905408 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.905342 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013"} err="failed to get container status \"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013\": rpc error: code = NotFound desc = could not find container \"42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013\": container with ID starting with 42adab76f93f0d9daf65080121381edefb8a8c24ac593315ee44ccfd35d7d013 not found: ID does not exist" Sep 18 22:24:24.915005 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.914968 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:24:24.917995 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:24.917964 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-027c8100/e2e-lifecycle-model-027c8100-kserve-f474955dc-dzftt"] Sep 18 22:24:25.524087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:25.524050 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:24:26.851432 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:26.851394 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" path="/var/lib/kubelet/pods/17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f/volumes" Sep 18 22:24:28.004239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:28.004197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:45.533290 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:45.533250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:47.279376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:47.279344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:54.478320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:54.478277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:54.599708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:54.599674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:54.872852 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:54.872720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:56.206849 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:56.206816 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:56.306259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:56.306224 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:56.418015 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:56.417983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:56.485723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:56.485639 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:24:57.252997 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:57.252964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:57.358336 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:57.358298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:24:58.712499 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:24:58.712469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:05.564080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:05.564040 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:05.691577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:05.691540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:06.347426 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:06.347393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:12.953368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:12.953336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:13.110202 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:13.110167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:13.216511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:13.216428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:17.233288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:17.233247 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:25:23.703176 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:23.703137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:23.914417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:23.914386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:37.554349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:37.554273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:38.110358 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:38.110323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:39.015398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:39.015364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:39.713498 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:39.713444 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:25:39.852512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:39.852481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:40.810829 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:40.810777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:41.821933 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:41.821882 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:25:41.832310 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:41.832269 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:25:41.846845 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:41.846803 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:25:41.855853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:41.855817 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:25:41.957414 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:41.957383 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:42.171012 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:42.170915 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:43.267315 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:43.267276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:44.265887 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:44.265842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:44.870341 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:44.870313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:47.198863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:47.198829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:52.402779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:52.402746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:52.619703 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:52.619667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:52.798339 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:52.798293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:53.183676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.183635 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:25:53.183960 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.183925 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" containerName="maas-api" containerID="cri-o://6f8faf59d24d2456abb60806366908024d03d1ec5b84d5a3df58e029531a7316" gracePeriod=30 Sep 18 22:25:53.315459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.315426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:53.373834 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.373780 2577 generic.go:356] "Generic (PLEG): container finished" podID="f76c7554-c4ad-4ca4-8fee-be5948f81176" containerID="6f8faf59d24d2456abb60806366908024d03d1ec5b84d5a3df58e029531a7316" exitCode=0 Sep 18 22:25:53.374020 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.373825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" event={"ID":"f76c7554-c4ad-4ca4-8fee-be5948f81176","Type":"ContainerDied","Data":"6f8faf59d24d2456abb60806366908024d03d1ec5b84d5a3df58e029531a7316"} Sep 18 22:25:53.486232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.486201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:25:53.574921 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.574889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:53.595688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.595653 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") pod \"f76c7554-c4ad-4ca4-8fee-be5948f81176\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " Sep 18 22:25:53.595688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.595693 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs\") pod \"f76c7554-c4ad-4ca4-8fee-be5948f81176\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " Sep 18 22:25:53.595983 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.595725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca\") pod \"f76c7554-c4ad-4ca4-8fee-be5948f81176\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " Sep 18 22:25:53.595983 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.595753 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg46d\" (UniqueName: \"kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d\") pod \"f76c7554-c4ad-4ca4-8fee-be5948f81176\" (UID: \"f76c7554-c4ad-4ca4-8fee-be5948f81176\") " Sep 18 22:25:53.596221 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.596190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f76c7554-c4ad-4ca4-8fee-be5948f81176" (UID: "f76c7554-c4ad-4ca4-8fee-be5948f81176"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:25:53.598348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.598315 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f76c7554-c4ad-4ca4-8fee-be5948f81176" (UID: "f76c7554-c4ad-4ca4-8fee-be5948f81176"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:53.598348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.598323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d" (OuterVolumeSpecName: "kube-api-access-wg46d") pod "f76c7554-c4ad-4ca4-8fee-be5948f81176" (UID: "f76c7554-c4ad-4ca4-8fee-be5948f81176"). InnerVolumeSpecName "kube-api-access-wg46d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:25:53.598517 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.598366 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f76c7554-c4ad-4ca4-8fee-be5948f81176" (UID: "f76c7554-c4ad-4ca4-8fee-be5948f81176"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:53.696926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.696888 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76c7554-c4ad-4ca4-8fee-be5948f81176-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:53.696926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.696921 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg46d\" (UniqueName: \"kubernetes.io/projected/f76c7554-c4ad-4ca4-8fee-be5948f81176-kube-api-access-wg46d\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:53.696926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.696930 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:53.696926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.696939 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76c7554-c4ad-4ca4-8fee-be5948f81176-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:53.756217 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.756096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:53.953919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:53.953881 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:54.150269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.150230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:54.354306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.354203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:54.381010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.380966 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" event={"ID":"f76c7554-c4ad-4ca4-8fee-be5948f81176","Type":"ContainerDied","Data":"4ef6b5021a3063d41e3a6b4afd93371c7179dc4eb5417eeb540d32bbdd1478b9"} Sep 18 22:25:54.381224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.380995 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk" Sep 18 22:25:54.381224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.381068 2577 scope.go:117] "RemoveContainer" containerID="6f8faf59d24d2456abb60806366908024d03d1ec5b84d5a3df58e029531a7316" Sep 18 22:25:54.410120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.410082 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:25:54.422348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.422304 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59b357-8567b7c679-rg8tk"] Sep 18 22:25:54.654125 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.654006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:54.849686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:54.849649 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" path="/var/lib/kubelet/pods/f76c7554-c4ad-4ca4-8fee-be5948f81176/volumes" Sep 18 22:25:55.650347 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:55.650312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:56.845917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:56.845874 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" podUID="2b6a536d-7f41-42dd-80f8-38da538c222f" containerName="main" containerID="cri-o://55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912" gracePeriod=45 Sep 18 22:25:56.853901 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:56.853861 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" podUID="a418aa2d-8df3-4977-98f4-60d36c0adb49" containerName="main" containerID="cri-o://5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3" gracePeriod=45 Sep 18 22:25:56.868978 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:56.868904 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" podUID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" containerName="main" containerID="cri-o://113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836" gracePeriod=45 Sep 18 22:25:56.885454 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:56.885409 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" podUID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" containerName="main" containerID="cri-o://09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f" gracePeriod=45 Sep 18 22:25:56.912811 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:56.912762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:57.038029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.037541 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:57.145362 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.145298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:57.175531 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.175487 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:25:57.201643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.201609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:25:57.216856 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.216820 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:25:57.244174 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.244139 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:25:57.339187 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339142 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.339390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.339390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339332 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.339390 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.339571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339396 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.339571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339441 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.339571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339481 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.339571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.339571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339545 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.339853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gfcl\" (UniqueName: \"kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.339853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339614 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.339853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339661 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9kmz\" (UniqueName: \"kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.339853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339707 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home" (OuterVolumeSpecName: "home") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.339853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339712 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwz2\" (UniqueName: \"kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.340139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.339866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djxjg\" (UniqueName: \"kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.340139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340002 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.340139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340011 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.340139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340097 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.340139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home\") pod \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\" (UID: \"ea1af7e6-2263-4fc1-a739-60e9815d7fec\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir\") pod \"a418aa2d-8df3-4977-98f4-60d36c0adb49\" (UID: \"a418aa2d-8df3-4977-98f4-60d36c0adb49\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340224 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340295 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340365 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm\") pod \"2b6a536d-7f41-42dd-80f8-38da538c222f\" (UID: \"2b6a536d-7f41-42dd-80f8-38da538c222f\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340389 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340423 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache\") pod \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\" (UID: \"8ecf3cd1-3498-4e6d-91bc-f9cecc022930\") " Sep 18 22:25:57.340480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340378 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home" (OuterVolumeSpecName: "home") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.341034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340863 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.341034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340772 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.341034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340885 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.341034 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.340899 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.341249 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.341127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.341249 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.341153 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.342574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.341390 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache" (OuterVolumeSpecName: "model-cache") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.342574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.341398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.342574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.341854 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.342574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.342256 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm" (OuterVolumeSpecName: "dshm") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.342574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.342287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home" (OuterVolumeSpecName: "home") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.343137 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.343070 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.343210 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.343159 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home" (OuterVolumeSpecName: "home") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.343477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.343452 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2" (OuterVolumeSpecName: "kube-api-access-4cwz2") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "kube-api-access-4cwz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:25:57.343576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.343494 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:57.343576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.343545 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:57.344080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.344040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl" (OuterVolumeSpecName: "kube-api-access-4gfcl") pod "a418aa2d-8df3-4977-98f4-60d36c0adb49" (UID: "a418aa2d-8df3-4977-98f4-60d36c0adb49"). InnerVolumeSpecName "kube-api-access-4gfcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:25:57.344476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.344445 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm" (OuterVolumeSpecName: "dshm") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.344741 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.344715 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm" (OuterVolumeSpecName: "dshm") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.345102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.345071 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm" (OuterVolumeSpecName: "dshm") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:25:57.345346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.345310 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b6a536d-7f41-42dd-80f8-38da538c222f" (UID: "2b6a536d-7f41-42dd-80f8-38da538c222f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:57.345911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.345890 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg" (OuterVolumeSpecName: "kube-api-access-djxjg") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "kube-api-access-djxjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:25:57.346066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.346041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ecf3cd1-3498-4e6d-91bc-f9cecc022930" (UID: "8ecf3cd1-3498-4e6d-91bc-f9cecc022930"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:25:57.346502 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.346479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz" (OuterVolumeSpecName: "kube-api-access-r9kmz") pod "ea1af7e6-2263-4fc1-a739-60e9815d7fec" (UID: "ea1af7e6-2263-4fc1-a739-60e9815d7fec"). InnerVolumeSpecName "kube-api-access-r9kmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:25:57.405661 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.405560 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" containerID="113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836" exitCode=0 Sep 18 22:25:57.405661 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.405609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" Sep 18 22:25:57.405944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.405654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" event={"ID":"ea1af7e6-2263-4fc1-a739-60e9815d7fec","Type":"ContainerDied","Data":"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836"} Sep 18 22:25:57.405944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.405708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv" event={"ID":"ea1af7e6-2263-4fc1-a739-60e9815d7fec","Type":"ContainerDied","Data":"4558eee109192350da289b3f4f4e334bf32637d2c2e898556c77e66b9c08d2c7"} Sep 18 22:25:57.405944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.405730 2577 scope.go:117] "RemoveContainer" containerID="113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836" Sep 18 22:25:57.407224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.407197 2577 generic.go:356] "Generic (PLEG): container finished" podID="a418aa2d-8df3-4977-98f4-60d36c0adb49" containerID="5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3" exitCode=0 Sep 18 22:25:57.407501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.407252 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" Sep 18 22:25:57.407501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.407296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" event={"ID":"a418aa2d-8df3-4977-98f4-60d36c0adb49","Type":"ContainerDied","Data":"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3"} Sep 18 22:25:57.407501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.407334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9" event={"ID":"a418aa2d-8df3-4977-98f4-60d36c0adb49","Type":"ContainerDied","Data":"670f7ab0fef07cbc3767fa0de58c5f3ae2b7d367e49b29ea0996032d21218f73"} Sep 18 22:25:57.408883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.408852 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b6a536d-7f41-42dd-80f8-38da538c222f" containerID="55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912" exitCode=0 Sep 18 22:25:57.409013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.408905 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" Sep 18 22:25:57.409013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.408917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" event={"ID":"2b6a536d-7f41-42dd-80f8-38da538c222f","Type":"ContainerDied","Data":"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912"} Sep 18 22:25:57.409013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.408956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp" event={"ID":"2b6a536d-7f41-42dd-80f8-38da538c222f","Type":"ContainerDied","Data":"3929b08220cf05ec19db770869b64b86b402c6b95c0757c0af30e1d19b5b4f2d"} Sep 18 22:25:57.410724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.410699 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" containerID="09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f" exitCode=0 Sep 18 22:25:57.410879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.410740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" Sep 18 22:25:57.410879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.410764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" event={"ID":"8ecf3cd1-3498-4e6d-91bc-f9cecc022930","Type":"ContainerDied","Data":"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f"} Sep 18 22:25:57.410879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.410823 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5" event={"ID":"8ecf3cd1-3498-4e6d-91bc-f9cecc022930","Type":"ContainerDied","Data":"6f82b33408472ff46120ed41267d07a2cc96b9d732c11447e3dd5eb376358d5a"} Sep 18 22:25:57.418304 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.418135 2577 scope.go:117] "RemoveContainer" containerID="113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836" Sep 18 22:25:57.418602 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:25:57.418574 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836\": container with ID starting with 113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836 not found: ID does not exist" containerID="113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836" Sep 18 22:25:57.418684 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.418620 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836"} err="failed to get container status \"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836\": rpc error: code = NotFound desc = could not find container \"113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836\": container with ID starting with 113d54aa2c5c01807c5297893b6fb9aa02b7d229ff15cbbd96d4adccb5dee836 not found: ID does not exist" Sep 18 22:25:57.418684 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.418646 2577 scope.go:117] "RemoveContainer" containerID="5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3" Sep 18 22:25:57.429335 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.429314 2577 scope.go:117] "RemoveContainer" containerID="5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3" Sep 18 22:25:57.429688 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:25:57.429669 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3\": container with ID starting with 5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3 not found: ID does not exist" containerID="5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3" Sep 18 22:25:57.429769 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.429696 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3"} err="failed to get container status \"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3\": rpc error: code = NotFound desc = could not find container \"5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3\": container with ID starting with 5a76f991eeee2bd765a063bdd30154382a5517a38045a8f8a79514ac84d5dfe3 not found: ID does not exist" Sep 18 22:25:57.429769 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.429716 2577 scope.go:117] "RemoveContainer" containerID="55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912" Sep 18 22:25:57.439323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.439302 2577 scope.go:117] "RemoveContainer" containerID="55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912" Sep 18 22:25:57.439664 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:25:57.439638 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912\": container with ID starting with 55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912 not found: ID does not exist" containerID="55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912" Sep 18 22:25:57.439760 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.439676 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912"} err="failed to get container status \"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912\": rpc error: code = NotFound desc = could not find container \"55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912\": container with ID starting with 55728ff952153712db67b71ac799bbebf43e52916a14e296f7459af01ed52912 not found: ID does not exist" Sep 18 22:25:57.439760 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.439702 2577 scope.go:117] "RemoveContainer" containerID="09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f" Sep 18 22:25:57.442238 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442214 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442238 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442240 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6a536d-7f41-42dd-80f8-38da538c222f-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442249 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfcl\" (UniqueName: \"kubernetes.io/projected/a418aa2d-8df3-4977-98f4-60d36c0adb49-kube-api-access-4gfcl\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442259 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442270 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kmz\" (UniqueName: \"kubernetes.io/projected/ea1af7e6-2263-4fc1-a739-60e9815d7fec-kube-api-access-r9kmz\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442278 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwz2\" (UniqueName: \"kubernetes.io/projected/2b6a536d-7f41-42dd-80f8-38da538c222f-kube-api-access-4cwz2\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442288 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxjg\" (UniqueName: \"kubernetes.io/projected/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-kube-api-access-djxjg\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442296 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442304 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442311 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1af7e6-2263-4fc1-a739-60e9815d7fec-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442319 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442325 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442332 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442339 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442346 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442353 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442359 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecf3cd1-3498-4e6d-91bc-f9cecc022930-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442366 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a418aa2d-8df3-4977-98f4-60d36c0adb49-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442374 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1af7e6-2263-4fc1-a739-60e9815d7fec-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442912 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442381 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a418aa2d-8df3-4977-98f4-60d36c0adb49-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.442912 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.442388 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6a536d-7f41-42dd-80f8-38da538c222f-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:25:57.449394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.449374 2577 scope.go:117] "RemoveContainer" containerID="09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f" Sep 18 22:25:57.449761 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:25:57.449725 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f\": container with ID starting with 09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f not found: ID does not exist" containerID="09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f" Sep 18 22:25:57.449913 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.449760 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f"} err="failed to get container status \"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f\": rpc error: code = NotFound desc = could not find container \"09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f\": container with ID starting with 09c85df413eea717483ad0ef9fd85495071d4e4442db822e42a3c254841ebf1f not found: ID does not exist" Sep 18 22:25:57.463322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.460325 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:25:57.467585 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.467544 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-2-w4-59b357-kserve-64c9b55bdf-qfmjp"] Sep 18 22:25:57.487386 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.487344 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:25:57.490633 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.490593 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/premium-w4-59b357-kserve-5785b8464c-qkjnv"] Sep 18 22:25:57.512105 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.512062 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:25:57.526563 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.526514 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/distinct-w4-59b357-kserve-6865dc8886-cfhr9"] Sep 18 22:25:57.541810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.541744 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:25:57.551425 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:57.551387 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59b357/simulator-w4-59b357-kserve-644f97ccd-n25n5"] Sep 18 22:25:58.599087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:58.599057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:25:58.848724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:58.848677 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6a536d-7f41-42dd-80f8-38da538c222f" path="/var/lib/kubelet/pods/2b6a536d-7f41-42dd-80f8-38da538c222f/volumes" Sep 18 22:25:58.849177 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:58.849132 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" path="/var/lib/kubelet/pods/8ecf3cd1-3498-4e6d-91bc-f9cecc022930/volumes" Sep 18 22:25:58.849492 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:58.849478 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a418aa2d-8df3-4977-98f4-60d36c0adb49" path="/var/lib/kubelet/pods/a418aa2d-8df3-4977-98f4-60d36c0adb49/volumes" Sep 18 22:25:58.850150 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:58.850134 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" path="/var/lib/kubelet/pods/ea1af7e6-2263-4fc1-a739-60e9815d7fec/volumes" Sep 18 22:25:59.552209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:25:59.552176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:00.214969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.214934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:26:00.560980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.560885 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:26:00.561287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561275 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561289 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" containerName="maas-api" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561303 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" containerName="maas-api" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561312 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561318 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561334 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a418aa2d-8df3-4977-98f4-60d36c0adb49" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561339 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a418aa2d-8df3-4977-98f4-60d36c0adb49" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561346 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6a536d-7f41-42dd-80f8-38da538c222f" containerName="main" Sep 18 22:26:00.561349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561351 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6a536d-7f41-42dd-80f8-38da538c222f" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561356 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561362 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561371 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" containerName="maas-api" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561376 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" containerName="maas-api" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561435 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76c7554-c4ad-4ca4-8fee-be5948f81176" containerName="maas-api" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561442 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecf3cd1-3498-4e6d-91bc-f9cecc022930" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561447 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17419ee1-7e92-4a0e-8e2f-ff0f1952fa2f" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561454 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b3a95c-3ead-4cdf-85e6-e70171fc093f" containerName="maas-api" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561459 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1af7e6-2263-4fc1-a739-60e9815d7fec" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561465 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a418aa2d-8df3-4977-98f4-60d36c0adb49" containerName="main" Sep 18 22:26:00.561641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.561471 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6a536d-7f41-42dd-80f8-38da538c222f" containerName="main" Sep 18 22:26:00.566527 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.566496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.577699 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.577675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4583f0-serving-cert\"" Sep 18 22:26:00.597014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.596979 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:26:00.666568 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.666536 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:00.669799 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.669752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.669919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.669809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.669974 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.669930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2h2\" (UniqueName: \"kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.670008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.669981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.770534 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.770450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.770773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.770550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.770773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.770653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2h2\" (UniqueName: \"kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.770773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.770712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.771016 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:26:00.770891 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-serving-cert: secret "maas-api-e2e-api-a-4583f0-serving-cert" not found Sep 18 22:26:00.771016 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:26:00.770960 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls podName:9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d nodeName:}" failed. No retries permitted until 2026-09-18 22:26:01.270938426 +0000 UTC m=+1546.942436380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls") pod "maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" (UID: "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d") : secret "maas-api-e2e-api-a-4583f0-serving-cert" not found Sep 18 22:26:00.771016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.770987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:00.771275 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.771255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.773376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.773351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.786196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.786161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2h2\" (UniqueName: \"kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:00.965683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:00.965647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:01.276354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.276238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:01.279071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.279044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") pod \"maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:01.478278 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.478221 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:01.775719 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.775684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:26:01.776750 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:26:01.776722 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8fe5ce_c406_4ec6_9929_b46e3f1ab76d.slice/crio-77c5473e552e98f7a41f52717fc2eb95554c160b4f2672349f454e6eda52e00e WatchSource:0}: Error finding container 77c5473e552e98f7a41f52717fc2eb95554c160b4f2672349f454e6eda52e00e: Status 404 returned error can't find the container with id 77c5473e552e98f7a41f52717fc2eb95554c160b4f2672349f454e6eda52e00e Sep 18 22:26:01.778281 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.778260 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:26:01.986561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:01.986532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:02.155712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.155672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:02.365171 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.365126 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:26:02.376078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.376039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.379305 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.379280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4583f0-dockercfg-mkk45\"" Sep 18 22:26:02.379503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.379480 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4583f0\"" Sep 18 22:26:02.392251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.392213 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:26:02.445749 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.445713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" event={"ID":"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d","Type":"ContainerStarted","Data":"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c"} Sep 18 22:26:02.445956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.445767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" event={"ID":"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d","Type":"ContainerStarted","Data":"77c5473e552e98f7a41f52717fc2eb95554c160b4f2672349f454e6eda52e00e"} Sep 18 22:26:02.445956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.445848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:02.474658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.474597 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" podStartSLOduration=2.03714943 podStartE2EDuration="2.474576839s" totalImagesPullingTime="437.427409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:02.472431267 +0000 UTC m=+1548.143929237" watchObservedRunningTime="2026-09-18 22:26:02.474576839 +0000 UTC m=+1548.146074835" Sep 18 22:26:02.488896 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.488854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.489113 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.488933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5zv\" (UniqueName: \"kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.590291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.590176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.590291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.590246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5zv\" (UniqueName: \"kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.591026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.591000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.602221 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.602190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5zv\" (UniqueName: \"kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv\") pod \"payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.687049 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.687014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:02.737674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.737642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:02.975393 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:02.975354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:26:02.976677 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:26:02.976648 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a50378_5f38_4d83_9fe9_d53ce8882e7d.slice/crio-0b477e4094f0d676c738f10010ffb7155451d52d1b49bb015c935cc471a36f93 WatchSource:0}: Error finding container 0b477e4094f0d676c738f10010ffb7155451d52d1b49bb015c935cc471a36f93: Status 404 returned error can't find the container with id 0b477e4094f0d676c738f10010ffb7155451d52d1b49bb015c935cc471a36f93 Sep 18 22:26:03.452394 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:03.452356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" event={"ID":"f9a50378-5f38-4d83-9fe9-d53ce8882e7d","Type":"ContainerStarted","Data":"0b477e4094f0d676c738f10010ffb7155451d52d1b49bb015c935cc471a36f93"} Sep 18 22:26:03.927600 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:03.927498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:04.079180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.079037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:04.237688 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.237656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:04.376577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.376520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:04.460568 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.460529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" event={"ID":"f9a50378-5f38-4d83-9fe9-d53ce8882e7d","Type":"ContainerStarted","Data":"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307"} Sep 18 22:26:04.460976 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.460600 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:04.528129 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:04.527999 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" podStartSLOduration=1.230804543 podStartE2EDuration="2.527976754s" totalImagesPullingTime="1.297172211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:04.526923946 +0000 UTC m=+1550.198421917" watchObservedRunningTime="2026-09-18 22:26:04.527976754 +0000 UTC m=+1550.199474702" Sep 18 22:26:05.454562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:05.454521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:08.458841 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:08.458806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:26:09.603998 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:09.603949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:14.446652 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:14.446621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:15.468965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:15.468923 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:26:18.171026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:18.170991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:18.271448 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:18.271417 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:20.315917 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:20.315863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:20.507979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:20.507922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:20.689520 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:20.689485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:20.788819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:20.788757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:22.684619 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:22.684588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:24.111590 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:24.111538 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:26:24.423606 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:24.423506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:25.237437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:25.237384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:26.394486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:26.394447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:26.918863 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:26.918833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:27.414865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:27.414831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:27.517930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:27.517898 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:27.748852 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:27.748821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:27.970970 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:27.970929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:28.077498 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:28.077400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:28.638585 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:28.638538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:29.621173 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:29.621137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:30.634729 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:30.634697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:31.164579 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:31.164546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:31.280526 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:31.280487 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:31.832561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:31.832527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:32.842639 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:32.842603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:33.925679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:33.925644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:35.466709 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:35.466677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:36.480115 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:36.480076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:37.599929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.599889 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:26:37.606728 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.606698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.609657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.609626 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b981e2-serving-cert\"" Sep 18 22:26:37.630292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.630253 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:26:37.682080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.682040 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:37.732909 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.732871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.732909 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.732920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw972\" (UniqueName: \"kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.733151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.733015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.733151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.733108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.800879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.800849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:37.833890 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.833850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.834048 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.833905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.834048 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.833932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw972\" (UniqueName: \"kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.834048 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.833995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.834219 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:26:37.834087 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-serving-cert: secret "maas-api-e2e-api-b-b981e2-serving-cert" not found Sep 18 22:26:37.834219 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:26:37.834153 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls podName:2d742a97-f999-4d5d-b1e9-f03620b2da01 nodeName:}" failed. No retries permitted until 2026-09-18 22:26:38.33413653 +0000 UTC m=+1584.005634478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls") pod "maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" (UID: "2d742a97-f999-4d5d-b1e9-f03620b2da01") : secret "maas-api-e2e-api-b-b981e2-serving-cert" not found Sep 18 22:26:37.834654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.834629 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.836857 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.836828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:37.852842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:37.852746 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw972\" (UniqueName: \"kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:38.005107 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.005074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:38.339836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.339777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:38.342532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.342496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") pod \"maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:38.446865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.446826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:38.521927 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.521890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:38.554698 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.554662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:38.622839 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.622724 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:26:38.628576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.628547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.631850 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.631814 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b981e2\"" Sep 18 22:26:38.632051 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.631857 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b981e2-dockercfg-cn9vv\"" Sep 18 22:26:38.647630 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.647591 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:26:38.712901 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.712861 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:26:38.717070 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.717046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.732670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.732636 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:26:38.744251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.744212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr22k\" (UniqueName: \"kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.744251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.744255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.816771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.816735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:26:38.818844 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:26:38.818815 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d742a97_f999_4d5d_b1e9_f03620b2da01.slice/crio-a1b2a728e77ce3db4c4cd3f0d82d3a9cbc0e2e21abd8485ede835da7ba0a38f5 WatchSource:0}: Error finding container a1b2a728e77ce3db4c4cd3f0d82d3a9cbc0e2e21abd8485ede835da7ba0a38f5: Status 404 returned error can't find the container with id a1b2a728e77ce3db4c4cd3f0d82d3a9cbc0e2e21abd8485ede835da7ba0a38f5 Sep 18 22:26:38.845065 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.845017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.845232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.845110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr22k\" (UniqueName: \"kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.845232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.845142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.845232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.845213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkmq\" (UniqueName: \"kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.846239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.846206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.856737 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.856703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr22k\" (UniqueName: \"kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k\") pod \"payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.940711 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.940666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:38.945861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.945828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.946066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.946042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkmq\" (UniqueName: \"kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.946653 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.946623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:38.955908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:38.955868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkmq\" (UniqueName: \"kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq\") pod \"payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:39.028536 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.028498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:39.224303 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.224273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:26:39.226986 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:26:39.226956 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386bf7a2_60b1_4cbb_a569_37c81506aba8.slice/crio-4e1bf7623991cd5ef651d9f4f64924ce23152b185a1e73dacadff1c97ed1c8f2 WatchSource:0}: Error finding container 4e1bf7623991cd5ef651d9f4f64924ce23152b185a1e73dacadff1c97ed1c8f2: Status 404 returned error can't find the container with id 4e1bf7623991cd5ef651d9f4f64924ce23152b185a1e73dacadff1c97ed1c8f2 Sep 18 22:26:39.316245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.316207 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:26:39.318661 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:26:39.318630 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a99331_aee3_4843_a9d8_a2611345188b.slice/crio-492dda07ec3e299ab511fca17e2c52ada92731babb2a2e9da0a21088b6526d37 WatchSource:0}: Error finding container 492dda07ec3e299ab511fca17e2c52ada92731babb2a2e9da0a21088b6526d37: Status 404 returned error can't find the container with id 492dda07ec3e299ab511fca17e2c52ada92731babb2a2e9da0a21088b6526d37 Sep 18 22:26:39.654270 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.654161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" event={"ID":"386bf7a2-60b1-4cbb-a569-37c81506aba8","Type":"ContainerStarted","Data":"f50e5c1f565d82f973c210b95ac157c3ad7dc09b3c54d198e6b1c1a407bc914a"} Sep 18 22:26:39.654861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.654828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" event={"ID":"386bf7a2-60b1-4cbb-a569-37c81506aba8","Type":"ContainerStarted","Data":"4e1bf7623991cd5ef651d9f4f64924ce23152b185a1e73dacadff1c97ed1c8f2"} Sep 18 22:26:39.656503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.656442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" event={"ID":"2d742a97-f999-4d5d-b1e9-f03620b2da01","Type":"ContainerStarted","Data":"b779ac679b36deba4ccf1dde9d74f756948d4efecbe54b85956e456e9f1b8b2f"} Sep 18 22:26:39.656503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.656505 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:39.656737 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.656529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" event={"ID":"2d742a97-f999-4d5d-b1e9-f03620b2da01","Type":"ContainerStarted","Data":"a1b2a728e77ce3db4c4cd3f0d82d3a9cbc0e2e21abd8485ede835da7ba0a38f5"} Sep 18 22:26:39.658372 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.658340 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" event={"ID":"99a99331-aee3-4843-a9d8-a2611345188b","Type":"ContainerStarted","Data":"201aa4fe289ebcce544f121f7a6400af3b88e9ddb93e829a691bd112f30b0bd8"} Sep 18 22:26:39.658537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.658392 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" event={"ID":"99a99331-aee3-4843-a9d8-a2611345188b","Type":"ContainerStarted","Data":"492dda07ec3e299ab511fca17e2c52ada92731babb2a2e9da0a21088b6526d37"} Sep 18 22:26:39.658537 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.658420 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:39.677601 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.677537 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" podStartSLOduration=1.6775161010000001 podStartE2EDuration="1.677516101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:39.674283321 +0000 UTC m=+1585.345781303" watchObservedRunningTime="2026-09-18 22:26:39.677516101 +0000 UTC m=+1585.349014071" Sep 18 22:26:39.697549 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.696927 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" podStartSLOduration=2.303017403 podStartE2EDuration="2.696904398s" totalImagesPullingTime="393.886995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:39.696550055 +0000 UTC m=+1585.368048016" watchObservedRunningTime="2026-09-18 22:26:39.696904398 +0000 UTC m=+1585.368402369" Sep 18 22:26:39.717140 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:39.717063 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" podStartSLOduration=1.7170406169999999 podStartE2EDuration="1.717040617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:39.714533537 +0000 UTC m=+1585.386031519" watchObservedRunningTime="2026-09-18 22:26:39.717040617 +0000 UTC m=+1585.388538620" Sep 18 22:26:40.666124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:40.666089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:26:43.612282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:43.612248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:26:44.686257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:44.686225 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:26:45.671099 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:45.671063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:26:50.667989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:50.667944 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:26:51.674167 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:26:51.674139 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:27:00.802518 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:00.802464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:00.915574 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:00.915526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:01.272942 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:01.272899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:02.052141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:02.052102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:03.218493 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:03.218454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:04.473268 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:04.473226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:04.506936 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:04.506900 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:27:05.422560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:05.422527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:06.302482 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:06.302395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:07.523996 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:07.523959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:08.561157 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:08.561123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:09.565972 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:09.565938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:10.662466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:10.662426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:10.722590 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:10.722553 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:27:10.774696 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:10.774656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:11.639878 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:11.639843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:11.966583 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:11.966554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:14.411011 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:14.410954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:16.398031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:16.397994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:16.480596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:16.480556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.108103 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.108068 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:27:20.108608 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.108338 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" podUID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" containerName="maas-api" containerID="cri-o://3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c" gracePeriod=30 Sep 18 22:27:20.144509 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.144476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.179349 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.179312 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:27:20.179570 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.179545 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" podUID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" containerName="payload-pre-processing" containerID="cri-o://944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307" gracePeriod=30 Sep 18 22:27:20.245244 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.245211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.349307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.348671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.442500 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.442427 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:27:20.455447 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.455415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.468320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.468291 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:27:20.567469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567429 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") pod \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " Sep 18 22:27:20.567679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567498 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs\") pod \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " Sep 18 22:27:20.567679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn5zv\" (UniqueName: \"kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv\") pod \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " Sep 18 22:27:20.567679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567602 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume\") pod \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\" (UID: \"f9a50378-5f38-4d83-9fe9-d53ce8882e7d\") " Sep 18 22:27:20.567679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567676 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca\") pod \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " Sep 18 22:27:20.567965 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.567703 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd2h2\" (UniqueName: \"kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2\") pod \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\" (UID: \"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d\") " Sep 18 22:27:20.568207 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.568163 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9a50378-5f38-4d83-9fe9-d53ce8882e7d" (UID: "f9a50378-5f38-4d83-9fe9-d53ce8882e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:27:20.568345 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.568248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" (UID: "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:27:20.570076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.570043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" (UID: "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:27:20.570198 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.570104 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv" (OuterVolumeSpecName: "kube-api-access-rn5zv") pod "f9a50378-5f38-4d83-9fe9-d53ce8882e7d" (UID: "f9a50378-5f38-4d83-9fe9-d53ce8882e7d"). InnerVolumeSpecName "kube-api-access-rn5zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:20.570491 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.570477 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" (UID: "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:27:20.570554 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.570506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2" (OuterVolumeSpecName: "kube-api-access-nd2h2") pod "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" (UID: "9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d"). InnerVolumeSpecName "kube-api-access-nd2h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:20.669465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669382 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn5zv\" (UniqueName: \"kubernetes.io/projected/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-kube-api-access-rn5zv\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.669465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669413 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a50378-5f38-4d83-9fe9-d53ce8882e7d-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.669465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669447 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.669465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669456 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2h2\" (UniqueName: \"kubernetes.io/projected/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-kube-api-access-nd2h2\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.669465 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669467 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.669748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.669476 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:20.787707 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.787673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:20.881135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.881094 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" containerID="3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c" exitCode=0 Sep 18 22:27:20.881135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.881143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" Sep 18 22:27:20.881396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.881177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" event={"ID":"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d","Type":"ContainerDied","Data":"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c"} Sep 18 22:27:20.881396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.881234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm" event={"ID":"9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d","Type":"ContainerDied","Data":"77c5473e552e98f7a41f52717fc2eb95554c160b4f2672349f454e6eda52e00e"} Sep 18 22:27:20.881396 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.881256 2577 scope.go:117] "RemoveContainer" containerID="3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c" Sep 18 22:27:20.882886 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.882867 2577 generic.go:356] "Generic (PLEG): container finished" podID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" containerID="944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307" exitCode=0 Sep 18 22:27:20.883016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.882910 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" Sep 18 22:27:20.883016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.882993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" event={"ID":"f9a50378-5f38-4d83-9fe9-d53ce8882e7d","Type":"ContainerDied","Data":"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307"} Sep 18 22:27:20.883110 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.883039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k" event={"ID":"f9a50378-5f38-4d83-9fe9-d53ce8882e7d","Type":"ContainerDied","Data":"0b477e4094f0d676c738f10010ffb7155451d52d1b49bb015c935cc471a36f93"} Sep 18 22:27:20.894203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.894182 2577 scope.go:117] "RemoveContainer" containerID="3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c" Sep 18 22:27:20.894571 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:27:20.894550 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c\": container with ID starting with 3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c not found: ID does not exist" containerID="3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c" Sep 18 22:27:20.894632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.894582 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c"} err="failed to get container status \"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c\": rpc error: code = NotFound desc = could not find container \"3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c\": container with ID starting with 3ec030f914892d498c762be9ffaade2042eea1b3423ce4c9a9ea72ac98ca4f8c not found: ID does not exist" Sep 18 22:27:20.894632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.894600 2577 scope.go:117] "RemoveContainer" containerID="944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307" Sep 18 22:27:20.904359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.904338 2577 scope.go:117] "RemoveContainer" containerID="944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307" Sep 18 22:27:20.904702 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:27:20.904683 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307\": container with ID starting with 944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307 not found: ID does not exist" containerID="944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307" Sep 18 22:27:20.904758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.904711 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307"} err="failed to get container status \"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307\": rpc error: code = NotFound desc = could not find container \"944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307\": container with ID starting with 944f9b6269cf57d20ee971013e05bd312dbcef9adcbaab17969f30f80f0e6307 not found: ID does not exist" Sep 18 22:27:20.905650 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.905628 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:27:20.908967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.908935 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4583f0-66ff448c64-xqdxm"] Sep 18 22:27:20.926395 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.926298 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:27:20.932225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.932169 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4583f0-5cfd4d5b5d-r5n4k"] Sep 18 22:27:20.991241 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:20.991205 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:21.895426 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:21.895394 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:22.848976 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:22.848940 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" path="/var/lib/kubelet/pods/9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d/volumes" Sep 18 22:27:22.849326 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:22.849311 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" path="/var/lib/kubelet/pods/f9a50378-5f38-4d83-9fe9-d53ce8882e7d/volumes" Sep 18 22:27:23.191373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:23.191339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:24.066760 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:24.066720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:25.194928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:25.194894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:25.743206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:25.743158 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" podUID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" containerName="main" containerID="cri-o://d4ae42b480dc0185d69eacd2b07c1422fcf5b3f68bb7fc4350b84994c8782ce6" gracePeriod=45 Sep 18 22:27:25.802453 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:25.802403 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:27:25.917408 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:25.917367 2577 generic.go:356] "Generic (PLEG): container finished" podID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" containerID="d4ae42b480dc0185d69eacd2b07c1422fcf5b3f68bb7fc4350b84994c8782ce6" exitCode=0 Sep 18 22:27:25.917560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:25.917438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" event={"ID":"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e","Type":"ContainerDied","Data":"d4ae42b480dc0185d69eacd2b07c1422fcf5b3f68bb7fc4350b84994c8782ce6"} Sep 18 22:27:26.007580 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.007504 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:27:26.121538 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121503 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxttb\" (UniqueName: \"kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.121743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121562 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.121743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121617 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.121743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121643 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.121743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121683 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.122008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121754 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs\") pod \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\" (UID: \"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e\") " Sep 18 22:27:26.122008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121952 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:27:26.122008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.121968 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home" (OuterVolumeSpecName: "home") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:27:26.122176 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.122010 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:27:26.122251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.122189 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.122251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.122216 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.122251 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.122228 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.124124 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.124089 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm" (OuterVolumeSpecName: "dshm") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:27:26.124255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.124212 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb" (OuterVolumeSpecName: "kube-api-access-pxttb") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "kube-api-access-pxttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:26.124316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.124268 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" (UID: "6fa44ea7-ddd9-40f8-aac2-a360fb7b077e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:27:26.223391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.223350 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.223391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.223385 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxttb\" (UniqueName: \"kubernetes.io/projected/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-kube-api-access-pxttb\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.223391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.223395 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:26.251392 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.251356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:26.923927 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.923889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" event={"ID":"6fa44ea7-ddd9-40f8-aac2-a360fb7b077e","Type":"ContainerDied","Data":"0d5588d3377ed54397f288310043c33ce2765ca1940dcbc787e73f6d8765ecdd"} Sep 18 22:27:26.923927 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.923917 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5" Sep 18 22:27:26.924182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.923942 2577 scope.go:117] "RemoveContainer" containerID="d4ae42b480dc0185d69eacd2b07c1422fcf5b3f68bb7fc4350b84994c8782ce6" Sep 18 22:27:26.945658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.945623 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:27:26.951108 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:26.951067 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/auth-test-model-123a30-kserve-59db7f8495-k7hw5"] Sep 18 22:27:28.440468 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:28.440433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:28.535906 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:28.535872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:28.684692 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:28.684651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:28.848589 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:28.848499 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" path="/var/lib/kubelet/pods/6fa44ea7-ddd9-40f8-aac2-a360fb7b077e/volumes" Sep 18 22:27:29.309245 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:29.309212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:30.128031 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:30.127997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:31.232443 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:31.232414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:32.026346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:32.026314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:32.963557 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:32.963524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:33.980891 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:33.980859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:34.937997 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:34.937963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:40.083104 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083068 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:27:40.083536 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083523 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" containerName="main" Sep 18 22:27:40.083598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083539 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" containerName="main" Sep 18 22:27:40.083598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083551 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" containerName="maas-api" Sep 18 22:27:40.083598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083556 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" containerName="maas-api" Sep 18 22:27:40.083598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083563 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" containerName="payload-pre-processing" Sep 18 22:27:40.083598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083570 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" containerName="payload-pre-processing" Sep 18 22:27:40.083770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083654 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8fe5ce-c406-4ec6-9929-b46e3f1ab76d" containerName="maas-api" Sep 18 22:27:40.083770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083668 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa44ea7-ddd9-40f8-aac2-a360fb7b077e" containerName="main" Sep 18 22:27:40.083770 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.083675 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a50378-5f38-4d83-9fe9-d53ce8882e7d" containerName="payload-pre-processing" Sep 18 22:27:40.088613 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.088583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.091673 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.091646 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"kube-root-ca.crt\"" Sep 18 22:27:40.092643 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.092623 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"openshift-service-ca.crt\"" Sep 18 22:27:40.092767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.092659 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"sub-test-model-123a30-kserve-self-signed-certs\"" Sep 18 22:27:40.092767 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.092681 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-123a30\"/\"default-dockercfg-jn7hd\"" Sep 18 22:27:40.107173 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.107140 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:27:40.128089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.128024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:40.161967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.161923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.161967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.161964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.162213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.162008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xjd\" (UniqueName: \"kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.162213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.162053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.162213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.162078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.162213 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.162103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.262989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.262941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xjd\" (UniqueName: \"kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.262989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.262995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263357 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263515 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263518 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.263651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.263598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.265654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.265628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.265971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.265950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.273377 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.273343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xjd\" (UniqueName: \"kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd\") pod \"sub-test-model-123a30-kserve-6d4749bc55-bjr2b\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.398523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.398423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:40.677609 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:40.677434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:27:40.680445 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:27:40.680392 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50cad7f2_2fc9_47aa_98d4_dd023feb88f2.slice/crio-37ad3cf907a13404f1de5f518c781fffcd349375f4a636bb7cd65b0fbdbe3797 WatchSource:0}: Error finding container 37ad3cf907a13404f1de5f518c781fffcd349375f4a636bb7cd65b0fbdbe3797: Status 404 returned error can't find the container with id 37ad3cf907a13404f1de5f518c781fffcd349375f4a636bb7cd65b0fbdbe3797 Sep 18 22:27:41.000576 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:41.000529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" event={"ID":"50cad7f2-2fc9-47aa-98d4-dd023feb88f2","Type":"ContainerStarted","Data":"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42"} Sep 18 22:27:41.000768 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:41.000587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" event={"ID":"50cad7f2-2fc9-47aa-98d4-dd023feb88f2","Type":"ContainerStarted","Data":"37ad3cf907a13404f1de5f518c781fffcd349375f4a636bb7cd65b0fbdbe3797"} Sep 18 22:27:41.020875 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:41.020821 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" podStartSLOduration=1.02078358 podStartE2EDuration="1.02078358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:27:41.01970804 +0000 UTC m=+1646.691206011" watchObservedRunningTime="2026-09-18 22:27:41.02078358 +0000 UTC m=+1646.692281550" Sep 18 22:27:44.526129 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.526089 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:27:44.737559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.737516 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:27:44.737817 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.737780 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" podUID="2d742a97-f999-4d5d-b1e9-f03620b2da01" containerName="maas-api" containerID="cri-o://b779ac679b36deba4ccf1dde9d74f756948d4efecbe54b85956e456e9f1b8b2f" gracePeriod=30 Sep 18 22:27:44.752969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.752934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:44.811219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.811113 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:27:44.811445 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.811407 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" podUID="99a99331-aee3-4843-a9d8-a2611345188b" containerName="payload-processing" containerID="cri-o://201aa4fe289ebcce544f121f7a6400af3b88e9ddb93e829a691bd112f30b0bd8" gracePeriod=30 Sep 18 22:27:44.835539 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.835505 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:27:44.835726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.835706 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" podUID="386bf7a2-60b1-4cbb-a569-37c81506aba8" containerName="payload-pre-processing" containerID="cri-o://f50e5c1f565d82f973c210b95ac157c3ad7dc09b3c54d198e6b1c1a407bc914a" gracePeriod=30 Sep 18 22:27:44.878102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.878066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:44.978626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:44.978598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:45.026091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.025838 2577 generic.go:356] "Generic (PLEG): container finished" podID="99a99331-aee3-4843-a9d8-a2611345188b" containerID="201aa4fe289ebcce544f121f7a6400af3b88e9ddb93e829a691bd112f30b0bd8" exitCode=0 Sep 18 22:27:45.026091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.026005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" event={"ID":"99a99331-aee3-4843-a9d8-a2611345188b","Type":"ContainerDied","Data":"201aa4fe289ebcce544f121f7a6400af3b88e9ddb93e829a691bd112f30b0bd8"} Sep 18 22:27:45.028641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.028610 2577 generic.go:356] "Generic (PLEG): container finished" podID="386bf7a2-60b1-4cbb-a569-37c81506aba8" containerID="f50e5c1f565d82f973c210b95ac157c3ad7dc09b3c54d198e6b1c1a407bc914a" exitCode=0 Sep 18 22:27:45.028847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.028679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" event={"ID":"386bf7a2-60b1-4cbb-a569-37c81506aba8","Type":"ContainerDied","Data":"f50e5c1f565d82f973c210b95ac157c3ad7dc09b3c54d198e6b1c1a407bc914a"} Sep 18 22:27:45.030728 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.030680 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d742a97-f999-4d5d-b1e9-f03620b2da01" containerID="b779ac679b36deba4ccf1dde9d74f756948d4efecbe54b85956e456e9f1b8b2f" exitCode=0 Sep 18 22:27:45.030926 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.030773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" event={"ID":"2d742a97-f999-4d5d-b1e9-f03620b2da01","Type":"ContainerDied","Data":"b779ac679b36deba4ccf1dde9d74f756948d4efecbe54b85956e456e9f1b8b2f"} Sep 18 22:27:45.087328 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.087291 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:27:45.114313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.114167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs\") pod \"2d742a97-f999-4d5d-b1e9-f03620b2da01\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " Sep 18 22:27:45.114583 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.114259 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca\") pod \"2d742a97-f999-4d5d-b1e9-f03620b2da01\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " Sep 18 22:27:45.115723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.114683 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") pod \"2d742a97-f999-4d5d-b1e9-f03620b2da01\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " Sep 18 22:27:45.115723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.114727 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw972\" (UniqueName: \"kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972\") pod \"2d742a97-f999-4d5d-b1e9-f03620b2da01\" (UID: \"2d742a97-f999-4d5d-b1e9-f03620b2da01\") " Sep 18 22:27:45.115723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.115141 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d742a97-f999-4d5d-b1e9-f03620b2da01" (UID: "2d742a97-f999-4d5d-b1e9-f03620b2da01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:27:45.118753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.118039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972" (OuterVolumeSpecName: "kube-api-access-dw972") pod "2d742a97-f999-4d5d-b1e9-f03620b2da01" (UID: "2d742a97-f999-4d5d-b1e9-f03620b2da01"). InnerVolumeSpecName "kube-api-access-dw972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:45.118753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.118126 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d742a97-f999-4d5d-b1e9-f03620b2da01" (UID: "2d742a97-f999-4d5d-b1e9-f03620b2da01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:27:45.118753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.118193 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2d742a97-f999-4d5d-b1e9-f03620b2da01" (UID: "2d742a97-f999-4d5d-b1e9-f03620b2da01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:27:45.119006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.118776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:45.128539 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.128510 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:27:45.137986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.137958 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:27:45.215919 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.215885 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume\") pod \"386bf7a2-60b1-4cbb-a569-37c81506aba8\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " Sep 18 22:27:45.216116 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.215969 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr22k\" (UniqueName: \"kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k\") pod \"386bf7a2-60b1-4cbb-a569-37c81506aba8\" (UID: \"386bf7a2-60b1-4cbb-a569-37c81506aba8\") " Sep 18 22:27:45.216116 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.215996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zkmq\" (UniqueName: \"kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq\") pod \"99a99331-aee3-4843-a9d8-a2611345188b\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " Sep 18 22:27:45.216116 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216083 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume\") pod \"99a99331-aee3-4843-a9d8-a2611345188b\" (UID: \"99a99331-aee3-4843-a9d8-a2611345188b\") " Sep 18 22:27:45.216314 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216303 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.216366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216321 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d742a97-f999-4d5d-b1e9-f03620b2da01-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.216366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216335 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d742a97-f999-4d5d-b1e9-f03620b2da01-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.216366 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216352 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw972\" (UniqueName: \"kubernetes.io/projected/2d742a97-f999-4d5d-b1e9-f03620b2da01-kube-api-access-dw972\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.216513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "386bf7a2-60b1-4cbb-a569-37c81506aba8" (UID: "386bf7a2-60b1-4cbb-a569-37c81506aba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:27:45.216619 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.216601 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99a99331-aee3-4843-a9d8-a2611345188b" (UID: "99a99331-aee3-4843-a9d8-a2611345188b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:27:45.218426 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.218398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq" (OuterVolumeSpecName: "kube-api-access-4zkmq") pod "99a99331-aee3-4843-a9d8-a2611345188b" (UID: "99a99331-aee3-4843-a9d8-a2611345188b"). InnerVolumeSpecName "kube-api-access-4zkmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:45.218898 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.218865 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k" (OuterVolumeSpecName: "kube-api-access-kr22k") pod "386bf7a2-60b1-4cbb-a569-37c81506aba8" (UID: "386bf7a2-60b1-4cbb-a569-37c81506aba8"). InnerVolumeSpecName "kube-api-access-kr22k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:27:45.228118 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.228088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:45.316985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.316883 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386bf7a2-60b1-4cbb-a569-37c81506aba8-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.316985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.316919 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr22k\" (UniqueName: \"kubernetes.io/projected/386bf7a2-60b1-4cbb-a569-37c81506aba8-kube-api-access-kr22k\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.316985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.316932 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkmq\" (UniqueName: \"kubernetes.io/projected/99a99331-aee3-4843-a9d8-a2611345188b-kube-api-access-4zkmq\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.316985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.316943 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a99331-aee3-4843-a9d8-a2611345188b-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:27:45.536194 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.536155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:45.639628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:45.639534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:46.037821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.037775 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" Sep 18 22:27:46.038035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.037816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s" event={"ID":"99a99331-aee3-4843-a9d8-a2611345188b","Type":"ContainerDied","Data":"492dda07ec3e299ab511fca17e2c52ada92731babb2a2e9da0a21088b6526d37"} Sep 18 22:27:46.038035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.037878 2577 scope.go:117] "RemoveContainer" containerID="201aa4fe289ebcce544f121f7a6400af3b88e9ddb93e829a691bd112f30b0bd8" Sep 18 22:27:46.039900 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.039422 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" Sep 18 22:27:46.039900 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.039419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng" event={"ID":"386bf7a2-60b1-4cbb-a569-37c81506aba8","Type":"ContainerDied","Data":"4e1bf7623991cd5ef651d9f4f64924ce23152b185a1e73dacadff1c97ed1c8f2"} Sep 18 22:27:46.041131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.041101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" event={"ID":"2d742a97-f999-4d5d-b1e9-f03620b2da01","Type":"ContainerDied","Data":"a1b2a728e77ce3db4c4cd3f0d82d3a9cbc0e2e21abd8485ede835da7ba0a38f5"} Sep 18 22:27:46.041131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.041129 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9" Sep 18 22:27:46.049215 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.049097 2577 scope.go:117] "RemoveContainer" containerID="f50e5c1f565d82f973c210b95ac157c3ad7dc09b3c54d198e6b1c1a407bc914a" Sep 18 22:27:46.058636 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.058611 2577 scope.go:117] "RemoveContainer" containerID="b779ac679b36deba4ccf1dde9d74f756948d4efecbe54b85956e456e9f1b8b2f" Sep 18 22:27:46.078244 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.078209 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:27:46.082569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.082536 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b981e2-bc886767d-zbkng"] Sep 18 22:27:46.101861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.101821 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:27:46.110581 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.110539 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b981e2-589bb497f4-9fjv9"] Sep 18 22:27:46.133634 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.133593 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:27:46.146549 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.146508 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b981e2-7cf8fb8467-6676s"] Sep 18 22:27:46.746999 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.746965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:46.849563 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.849527 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d742a97-f999-4d5d-b1e9-f03620b2da01" path="/var/lib/kubelet/pods/2d742a97-f999-4d5d-b1e9-f03620b2da01/volumes" Sep 18 22:27:46.849933 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.849919 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386bf7a2-60b1-4cbb-a569-37c81506aba8" path="/var/lib/kubelet/pods/386bf7a2-60b1-4cbb-a569-37c81506aba8/volumes" Sep 18 22:27:46.850240 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:46.850229 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a99331-aee3-4843-a9d8-a2611345188b" path="/var/lib/kubelet/pods/99a99331-aee3-4843-a9d8-a2611345188b/volumes" Sep 18 22:27:48.048392 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:48.048361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:49.000988 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:49.000950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:49.883484 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:49.883449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:50.399356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:50.399317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:50.399356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:50.399360 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:50.412503 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:50.412470 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:51.086936 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:51.086898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:27:52.155403 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:52.155371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:52.308046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:52.307985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:52.469862 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:52.469775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:53.124908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:53.124867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:54.134959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:54.134923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:55.260578 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:55.260549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:56.032224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:56.032194 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:57.031409 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:57.031371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:58.129912 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:58.129877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:27:59.113027 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:27:59.112986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:28:03.631629 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:03.631593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:03.792572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:03.792538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:04.095616 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:04.095581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:04.244821 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:04.244762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:04.562397 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:04.562364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:04.741215 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:04.741177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:05.632951 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:05.632920 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:06.506944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:06.506913 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:07.288010 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.287974 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:28:07.292859 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.292823 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:28:07.293248 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293235 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d742a97-f999-4d5d-b1e9-f03620b2da01" containerName="maas-api" Sep 18 22:28:07.293296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293251 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d742a97-f999-4d5d-b1e9-f03620b2da01" containerName="maas-api" Sep 18 22:28:07.293296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a99331-aee3-4843-a9d8-a2611345188b" containerName="payload-processing" Sep 18 22:28:07.293296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293268 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a99331-aee3-4843-a9d8-a2611345188b" containerName="payload-processing" Sep 18 22:28:07.293296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386bf7a2-60b1-4cbb-a569-37c81506aba8" containerName="payload-pre-processing" Sep 18 22:28:07.293296 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293288 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="386bf7a2-60b1-4cbb-a569-37c81506aba8" containerName="payload-pre-processing" Sep 18 22:28:07.293453 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293354 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a99331-aee3-4843-a9d8-a2611345188b" containerName="payload-processing" Sep 18 22:28:07.293453 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293364 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d742a97-f999-4d5d-b1e9-f03620b2da01" containerName="maas-api" Sep 18 22:28:07.293453 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.293369 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="386bf7a2-60b1-4cbb-a569-37c81506aba8" containerName="payload-pre-processing" Sep 18 22:28:07.306093 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.306051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.314181 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.314157 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"kube-root-ca.crt\"" Sep 18 22:28:07.314761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.314743 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"default-dockercfg-hptzx\"" Sep 18 22:28:07.314864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.314804 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"openshift-service-ca.crt\"" Sep 18 22:28:07.315334 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.315319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"sub-test-model-7f10d8-kserve-self-signed-certs\"" Sep 18 22:28:07.319620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.319593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:28:07.338819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.338763 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:07.428713 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.428675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.428713 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.428716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.428975 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.428752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.428975 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.428962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8n4\" (UniqueName: \"kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.429043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.429012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.429077 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.429056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.529904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.529863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8n4\" (UniqueName: \"kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.529913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.530522 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.530503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.532562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.532539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.532959 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.532936 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.539577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.539503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8n4\" (UniqueName: \"kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4\") pod \"sub-test-model-7f10d8-kserve-6454668879-rhdqm\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.616436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.616394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:07.647065 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.647021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:07.905345 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:07.905277 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:28:07.908082 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:28:07.908053 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a0d2e9_b4f8_49b8_8994_e27e3dcafdd3.slice/crio-ab70ff0e30c344d8f5e5d04753fd82ea0b3744cb1bbc666976b18c7cfd36407d WatchSource:0}: Error finding container ab70ff0e30c344d8f5e5d04753fd82ea0b3744cb1bbc666976b18c7cfd36407d: Status 404 returned error can't find the container with id ab70ff0e30c344d8f5e5d04753fd82ea0b3744cb1bbc666976b18c7cfd36407d Sep 18 22:28:08.169232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:08.169132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" event={"ID":"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3","Type":"ContainerStarted","Data":"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061"} Sep 18 22:28:08.169232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:08.169177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" event={"ID":"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3","Type":"ContainerStarted","Data":"ab70ff0e30c344d8f5e5d04753fd82ea0b3744cb1bbc666976b18c7cfd36407d"} Sep 18 22:28:08.192590 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:08.192526 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" podStartSLOduration=1.192507246 podStartE2EDuration="1.192507246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:28:08.192057268 +0000 UTC m=+1673.863555298" watchObservedRunningTime="2026-09-18 22:28:08.192507246 +0000 UTC m=+1673.864005272" Sep 18 22:28:11.381820 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:11.381760 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:11.494720 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:11.494680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:11.724726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:11.724690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:12.359854 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:12.359819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:13.632470 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:13.632441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:14.467641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:14.467607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:15.396033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:15.395917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:16.623006 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:16.622969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:17.617030 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:17.616993 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:17.617243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:17.617049 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:17.631181 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:17.631145 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:18.232391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:18.232356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:28:22.745028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:22.744990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:22.878472 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:22.878439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:22.982101 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:22.982069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:23.077538 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:23.077454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:23.546654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:23.546613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:23.664350 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:23.664307 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:28:23.679196 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:23.679161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:24.728405 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:24.728372 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:25.705844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:25.705814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:29.788047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:29.788002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:30.002984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:30.002950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:30.141583 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:30.141490 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:35.017180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:35.017136 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:28:47.918572 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:47.918485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:28:52.005429 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:52.005388 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:28:52.044927 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:52.044890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:53.077243 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:53.077184 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:28:53.109203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:53.109171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:28:57.452866 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:28:57.452806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:29:01.114141 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:01.114102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:29:07.025633 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.025583 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" podUID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" containerName="main" containerID="cri-o://809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42" gracePeriod=45 Sep 18 22:29:07.280154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.280080 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:29:07.327604 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327564 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.327825 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327619 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.327825 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327648 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.327825 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327667 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2xjd\" (UniqueName: \"kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.327825 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327699 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.327825 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.327761 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home\") pod \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\" (UID: \"50cad7f2-2fc9-47aa-98d4-dd023feb88f2\") " Sep 18 22:29:07.328126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:07.328126 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328050 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:07.328219 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328132 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home" (OuterVolumeSpecName: "home") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:07.328259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328220 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.328259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328239 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.328259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.328254 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.330087 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.330062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm" (OuterVolumeSpecName: "dshm") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:07.330169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.330099 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd" (OuterVolumeSpecName: "kube-api-access-f2xjd") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "kube-api-access-f2xjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:29:07.330169 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.330146 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50cad7f2-2fc9-47aa-98d4-dd023feb88f2" (UID: "50cad7f2-2fc9-47aa-98d4-dd023feb88f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:29:07.429596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.429539 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.429596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.429584 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.429596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.429595 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xjd\" (UniqueName: \"kubernetes.io/projected/50cad7f2-2fc9-47aa-98d4-dd023feb88f2-kube-api-access-f2xjd\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:07.482256 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.482217 2577 generic.go:356] "Generic (PLEG): container finished" podID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" containerID="809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42" exitCode=0 Sep 18 22:29:07.482437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.482263 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" Sep 18 22:29:07.482437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.482278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" event={"ID":"50cad7f2-2fc9-47aa-98d4-dd023feb88f2","Type":"ContainerDied","Data":"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42"} Sep 18 22:29:07.482437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.482318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b" event={"ID":"50cad7f2-2fc9-47aa-98d4-dd023feb88f2","Type":"ContainerDied","Data":"37ad3cf907a13404f1de5f518c781fffcd349375f4a636bb7cd65b0fbdbe3797"} Sep 18 22:29:07.482437 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.482339 2577 scope.go:117] "RemoveContainer" containerID="809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42" Sep 18 22:29:07.494664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.494637 2577 scope.go:117] "RemoveContainer" containerID="809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42" Sep 18 22:29:07.495053 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:29:07.495032 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42\": container with ID starting with 809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42 not found: ID does not exist" containerID="809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42" Sep 18 22:29:07.495128 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.495062 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42"} err="failed to get container status \"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42\": rpc error: code = NotFound desc = could not find container \"809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42\": container with ID starting with 809de39835b07ab4b1cf12638647e19cabbb1b1251fc6258cf41dcc757056a42 not found: ID does not exist" Sep 18 22:29:07.507647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.507615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:29:07.512233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:07.512199 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-123a30/sub-test-model-123a30-kserve-6d4749bc55-bjr2b"] Sep 18 22:29:08.097575 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.097529 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerName="main" containerID="cri-o://905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061" gracePeriod=45 Sep 18 22:29:08.222072 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.222033 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused" Sep 18 22:29:08.362884 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.362780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:29:08.440506 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440464 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.440506 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440506 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.440779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg8n4\" (UniqueName: \"kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.440779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440576 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.440779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440606 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.440779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440661 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs\") pod \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\" (UID: \"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3\") " Sep 18 22:29:08.441044 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:08.441044 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440926 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:08.441044 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.440947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home" (OuterVolumeSpecName: "home") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:08.441203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.441111 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.441203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.441130 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.441203 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.441140 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.443016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.442990 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4" (OuterVolumeSpecName: "kube-api-access-pg8n4") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "kube-api-access-pg8n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:29:08.443097 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.443053 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm" (OuterVolumeSpecName: "dshm") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:29:08.443147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.443107 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" (UID: "69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:29:08.488716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.488676 2577 generic.go:356] "Generic (PLEG): container finished" podID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerID="905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061" exitCode=0 Sep 18 22:29:08.488925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.488723 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" Sep 18 22:29:08.488925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.488761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" event={"ID":"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3","Type":"ContainerDied","Data":"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061"} Sep 18 22:29:08.488925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.488822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm" event={"ID":"69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3","Type":"ContainerDied","Data":"ab70ff0e30c344d8f5e5d04753fd82ea0b3744cb1bbc666976b18c7cfd36407d"} Sep 18 22:29:08.488925 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.488842 2577 scope.go:117] "RemoveContainer" containerID="905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061" Sep 18 22:29:08.498607 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.498587 2577 scope.go:117] "RemoveContainer" containerID="905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061" Sep 18 22:29:08.498963 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:29:08.498943 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061\": container with ID starting with 905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061 not found: ID does not exist" containerID="905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061" Sep 18 22:29:08.499028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.498972 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061"} err="failed to get container status \"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061\": rpc error: code = NotFound desc = could not find container \"905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061\": container with ID starting with 905c6e08fbd3a7a593d8dcc49d26c861682ca8e00592624c175881558d1ec061 not found: ID does not exist" Sep 18 22:29:08.516367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.516326 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:29:08.525753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.525719 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/sub-test-model-7f10d8-kserve-6454668879-rhdqm"] Sep 18 22:29:08.542559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.542531 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.542559 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.542558 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8n4\" (UniqueName: \"kubernetes.io/projected/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-kube-api-access-pg8n4\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.542716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.542568 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:29:08.849032 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.848993 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" path="/var/lib/kubelet/pods/50cad7f2-2fc9-47aa-98d4-dd023feb88f2/volumes" Sep 18 22:29:08.849398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:08.849385 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" path="/var/lib/kubelet/pods/69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3/volumes" Sep 18 22:29:20.109329 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.109278 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:29:20.109907 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.109888 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerName="main" Sep 18 22:29:20.110001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.109910 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerName="main" Sep 18 22:29:20.110001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.109932 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" containerName="main" Sep 18 22:29:20.110001 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.109940 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" containerName="main" Sep 18 22:29:20.110158 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.110037 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cad7f2-2fc9-47aa-98d4-dd023feb88f2" containerName="main" Sep 18 22:29:20.110158 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.110051 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a0d2e9-b4f8-49b8-8994-e27e3dcafdd3" containerName="main" Sep 18 22:29:20.113312 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.113285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.128024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.127991 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"openshift-service-ca.crt\"" Sep 18 22:29:20.128239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.128029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"kube-root-ca.crt\"" Sep 18 22:29:20.128239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.127993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"rate-test-model-7f10d8-kserve-self-signed-certs\"" Sep 18 22:29:20.128239 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.127991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7f10d8\"/\"default-dockercfg-hptzx\"" Sep 18 22:29:20.157726 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.157682 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:29:20.167189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.167157 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:29:20.254924 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.254879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.255146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.254937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.255146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.255026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.255146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.255060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.255146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.255139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.255342 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.255220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457rn\" (UniqueName: \"kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356508 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457rn\" (UniqueName: \"kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.356724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.356673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.357067 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.357044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.357136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.357082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.357136 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.357118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.359292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.359269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.359553 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.359510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.373486 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.373448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-457rn\" (UniqueName: \"kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn\") pod \"rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.422774 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.422730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:20.715227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:20.715196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:29:20.716920 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:29:20.716892 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab509193_5b8b_4d3e_bbf2_dd7a58e2f878.slice/crio-119dfc45205727cfd4a76fde98e6d94d11dbcdc3f8e39eac3d35bee935fd966b WatchSource:0}: Error finding container 119dfc45205727cfd4a76fde98e6d94d11dbcdc3f8e39eac3d35bee935fd966b: Status 404 returned error can't find the container with id 119dfc45205727cfd4a76fde98e6d94d11dbcdc3f8e39eac3d35bee935fd966b Sep 18 22:29:21.556201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:21.556161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" event={"ID":"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878","Type":"ContainerStarted","Data":"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30"} Sep 18 22:29:21.556201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:21.556208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" event={"ID":"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878","Type":"ContainerStarted","Data":"119dfc45205727cfd4a76fde98e6d94d11dbcdc3f8e39eac3d35bee935fd966b"} Sep 18 22:29:21.586111 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:21.586060 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" podStartSLOduration=1.5860418680000001 podStartE2EDuration="1.586041868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:21.585216437 +0000 UTC m=+1747.256714407" watchObservedRunningTime="2026-09-18 22:29:21.586041868 +0000 UTC m=+1747.257539867" Sep 18 22:29:23.968283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:23.968238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:29:30.423184 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:30.423122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:30.423744 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:30.423213 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:30.436311 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:30.436283 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:30.625030 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:30.624992 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:29:39.397743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:39.397707 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:29:39.646979 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:39.646942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:29:40.964186 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:40.964143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:29:59.252626 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:29:59.252581 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:30:05.154838 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:05.154799 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:30:09.466890 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:09.466801 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:30:10.922176 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:10.922145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:11.034214 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:11.034179 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:12.590658 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:12.590626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:12.826198 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:12.826161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:15.980879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:15.980843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:16.185139 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:16.185103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:16.758985 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:16.758941 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:30:17.130369 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:17.130279 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:19.129325 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:19.129289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:20.175398 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.175331 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" podUID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" containerName="main" containerID="cri-o://825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30" gracePeriod=45 Sep 18 22:30:20.436295 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.436253 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:30:20.540773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540801 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540832 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-457rn\" (UniqueName: \"kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540894 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.540953 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm\") pod \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\" (UID: \"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878\") " Sep 18 22:30:20.541316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.541061 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:30:20.541316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.541095 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home" (OuterVolumeSpecName: "home") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:30:20.541316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.541267 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.541316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.541266 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:30:20.541316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.541287 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.543264 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.543230 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:30:20.543264 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.543246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm" (OuterVolumeSpecName: "dshm") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:30:20.543431 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.543309 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn" (OuterVolumeSpecName: "kube-api-access-457rn") pod "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" (UID: "ab509193-5b8b-4d3e-bbf2-dd7a58e2f878"). InnerVolumeSpecName "kube-api-access-457rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:30:20.642002 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.641961 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-457rn\" (UniqueName: \"kubernetes.io/projected/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-kube-api-access-457rn\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.642002 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.641998 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.642002 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.642008 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.642320 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.642016 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:30:20.706023 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.705937 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:20.861565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.861534 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" containerID="825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30" exitCode=0 Sep 18 22:30:20.861756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.861584 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" Sep 18 22:30:20.861756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.861620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" event={"ID":"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878","Type":"ContainerDied","Data":"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30"} Sep 18 22:30:20.861756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.861666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm" event={"ID":"ab509193-5b8b-4d3e-bbf2-dd7a58e2f878","Type":"ContainerDied","Data":"119dfc45205727cfd4a76fde98e6d94d11dbcdc3f8e39eac3d35bee935fd966b"} Sep 18 22:30:20.861756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.861686 2577 scope.go:117] "RemoveContainer" containerID="825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30" Sep 18 22:30:20.871111 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.871089 2577 scope.go:117] "RemoveContainer" containerID="825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30" Sep 18 22:30:20.871443 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:30:20.871420 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30\": container with ID starting with 825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30 not found: ID does not exist" containerID="825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30" Sep 18 22:30:20.871497 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.871453 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30"} err="failed to get container status \"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30\": rpc error: code = NotFound desc = could not find container \"825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30\": container with ID starting with 825b4bf1b22bea303545a2b3d940abdc4a54b04d82cef989e82aff848dfdcd30 not found: ID does not exist" Sep 18 22:30:20.923639 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.923602 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:30:20.952317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:20.952276 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7f10d8/rate-test-model-7f10d8-kserve-6c97c8fd9f-4ggpm"] Sep 18 22:30:21.086907 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:21.086824 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:22.107457 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.107412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:22.225336 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.225305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:22.291920 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.291883 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:30:22.292293 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.292281 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" containerName="main" Sep 18 22:30:22.292340 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.292296 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" containerName="main" Sep 18 22:30:22.292382 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.292373 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" containerName="main" Sep 18 22:30:22.296929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.296899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.299405 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.299375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-adba81\"" Sep 18 22:30:22.319283 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.319250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:30:22.357986 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.357899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.358147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.357984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9v7\" (UniqueName: \"kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.458840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.458767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9v7\" (UniqueName: \"kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.459029 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.458917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.459621 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.459602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.472773 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.472740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9v7\" (UniqueName: \"kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7\") pod \"payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.608570 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.608478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:22.849450 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.849416 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab509193-5b8b-4d3e-bbf2-dd7a58e2f878" path="/var/lib/kubelet/pods/ab509193-5b8b-4d3e-bbf2-dd7a58e2f878/volumes" Sep 18 22:30:22.923966 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:22.923912 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:30:22.926940 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:30:22.926914 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4d6288_8437_4f22_b13d_59affaa5b64c.slice/crio-cacfdd840be39a5c53a884edb04e0252a8a795639114bd44ac803f4f22b6424a WatchSource:0}: Error finding container cacfdd840be39a5c53a884edb04e0252a8a795639114bd44ac803f4f22b6424a: Status 404 returned error can't find the container with id cacfdd840be39a5c53a884edb04e0252a8a795639114bd44ac803f4f22b6424a Sep 18 22:30:23.879026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:23.878991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" event={"ID":"ec4d6288-8437-4f22-b13d-59affaa5b64c","Type":"ContainerStarted","Data":"01ea5517546e318e3ed9ab9ef4e5e21c74e75626c7044be6a568247bd4f86e1f"} Sep 18 22:30:23.879454 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:23.879044 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" event={"ID":"ec4d6288-8437-4f22-b13d-59affaa5b64c","Type":"ContainerStarted","Data":"cacfdd840be39a5c53a884edb04e0252a8a795639114bd44ac803f4f22b6424a"} Sep 18 22:30:23.879454 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:23.879118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:23.911373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:23.911322 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" podStartSLOduration=1.9113046790000001 podStartE2EDuration="1.911304679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:23.909931216 +0000 UTC m=+1809.581429187" watchObservedRunningTime="2026-09-18 22:30:23.911304679 +0000 UTC m=+1809.582802648" Sep 18 22:30:26.245780 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:26.245739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:30:34.886464 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:34.886426 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:30:44.254476 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:44.254444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:45.088313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:45.088266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:45.957410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:45.957364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:46.059573 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:46.059538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:46.820714 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:46.820665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:47.163265 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:47.163170 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:47.544801 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:47.544759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:47.791548 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:47.791510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:49.292026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:49.291987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:49.929066 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:49.929035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:50.024420 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:50.024382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:50.354469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:50.354422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:51.422969 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:51.422927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:52.320287 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:52.320240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:53.191337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.191299 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:30:53.195024 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.194995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.197651 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.197625 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-58a8ca-serving-cert\"" Sep 18 22:30:53.205907 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.205866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:30:53.224620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.224585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:53.269598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.269557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4qv\" (UniqueName: \"kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.269598 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.269598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.269944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.269752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.269944 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.269851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.289179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.289145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:53.370386 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.370347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.370566 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.370395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.370566 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.370433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4qv\" (UniqueName: \"kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.370566 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.370456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.370566 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:30:53.370520 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-serving-cert: secret "maas-api-e2e-ipp-b-58a8ca-serving-cert" not found Sep 18 22:30:53.370777 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:30:53.370613 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls podName:baa63048-1317-41f1-8a55-cd6b514f9218 nodeName:}" failed. No retries permitted until 2026-09-18 22:30:53.870589255 +0000 UTC m=+1839.542087222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls") pod "maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" (UID: "baa63048-1317-41f1-8a55-cd6b514f9218") : secret "maas-api-e2e-ipp-b-58a8ca-serving-cert" not found Sep 18 22:30:53.371292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.371261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.373389 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.373362 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.379841 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.379814 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4qv\" (UniqueName: \"kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.461147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.461058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:53.839840 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.839738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:53.874724 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.874678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.877520 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.877483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") pod \"maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:53.942842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:53.942698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:30:54.020271 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.020237 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:30:54.024201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.024170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.027019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.026993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-58a8ca-dockercfg-ddbnm\"" Sep 18 22:30:54.027830 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.027809 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-58a8ca\"" Sep 18 22:30:54.035262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.035217 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:30:54.076346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.076309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjnjs\" (UniqueName: \"kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.076553 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.076369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.110194 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.110113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:54.178026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.177579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjnjs\" (UniqueName: \"kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.178026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.177662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.178553 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.178528 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.196596 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.196564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjnjs\" (UniqueName: \"kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs\") pod \"payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.337662 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.337612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:54.393146 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.393122 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:30:54.625205 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:54.625177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:30:54.627127 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:30:54.627082 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b061bc_1cc8_4218_8bb0_ace8c951d633.slice/crio-cf13260cb817e0cc3fcd2c8a087170f8b8af3741986f23ae496cb7198668b3b0 WatchSource:0}: Error finding container cf13260cb817e0cc3fcd2c8a087170f8b8af3741986f23ae496cb7198668b3b0: Status 404 returned error can't find the container with id cf13260cb817e0cc3fcd2c8a087170f8b8af3741986f23ae496cb7198668b3b0 Sep 18 22:30:55.048847 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.048800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" event={"ID":"baa63048-1317-41f1-8a55-cd6b514f9218","Type":"ContainerStarted","Data":"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76"} Sep 18 22:30:55.049071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.048865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" event={"ID":"baa63048-1317-41f1-8a55-cd6b514f9218","Type":"ContainerStarted","Data":"ac67db889e73aa58b48917760a16175a82b196bb3a4d95ffd52f2f66642aef28"} Sep 18 22:30:55.049071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.048987 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:30:55.050683 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.050648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" event={"ID":"c7b061bc-1cc8-4218-8bb0-ace8c951d633","Type":"ContainerStarted","Data":"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a"} Sep 18 22:30:55.050861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.050695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" event={"ID":"c7b061bc-1cc8-4218-8bb0-ace8c951d633","Type":"ContainerStarted","Data":"cf13260cb817e0cc3fcd2c8a087170f8b8af3741986f23ae496cb7198668b3b0"} Sep 18 22:30:55.050861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.050741 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:30:55.067567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.067508 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" podStartSLOduration=1.613366621 podStartE2EDuration="2.067488692s" totalImagesPullingTime="454.122071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:55.066240421 +0000 UTC m=+1840.737738390" watchObservedRunningTime="2026-09-18 22:30:55.067488692 +0000 UTC m=+1840.738986663" Sep 18 22:30:55.087428 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:55.087362 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" podStartSLOduration=1.087337398 podStartE2EDuration="1.087337398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:55.085222769 +0000 UTC m=+1840.756720739" watchObservedRunningTime="2026-09-18 22:30:55.087337398 +0000 UTC m=+1840.758835369" Sep 18 22:30:57.219242 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:30:57.219201 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:31:01.061687 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:01.061652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:31:06.057902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:06.057869 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:31:15.749167 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:15.749127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:17.110763 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:17.110709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:17.559359 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:17.559324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:18.509992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:18.509953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:19.281856 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:19.281816 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:20.048782 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:20.048735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:34.027924 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.027890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:34.722467 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.722430 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:31:34.726412 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.726387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.730507 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.730476 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-adba81\"/\"kube-root-ca.crt\"" Sep 18 22:31:34.731201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.731180 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-adba81\"/\"default-dockercfg-nktzw\"" Sep 18 22:31:34.731330 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.731213 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-adba81\"/\"openshift-service-ca.crt\"" Sep 18 22:31:34.731330 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.731272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-adba81\"/\"ipp-route-adba81-kserve-self-signed-certs\"" Sep 18 22:31:34.780101 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.780101 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.780363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.780363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.780363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.780363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.780316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8bg\" (UniqueName: \"kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.793931 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.793896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:31:34.799534 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.799492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:31:34.881470 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881470 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8bg\" (UniqueName: \"kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881937 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.881996 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.882056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.881996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.884374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.884337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.884514 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.884472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:34.898073 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:34.898041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8bg\" (UniqueName: \"kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg\") pod \"ipp-route-adba81-kserve-69598bf76c-49qml\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:35.036837 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:35.036705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:35.327270 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:35.327234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:31:35.329436 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:31:35.329405 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d4b1359_4942_4ace_8cae_9cb648e2cf6c.slice/crio-a1d98573bdb254eb08f747148f0038db1ce0e42b7430e2c204ec4c4c4a981023 WatchSource:0}: Error finding container a1d98573bdb254eb08f747148f0038db1ce0e42b7430e2c204ec4c4c4a981023: Status 404 returned error can't find the container with id a1d98573bdb254eb08f747148f0038db1ce0e42b7430e2c204ec4c4c4a981023 Sep 18 22:31:36.275000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:36.274912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" event={"ID":"5d4b1359-4942-4ace-8cae-9cb648e2cf6c","Type":"ContainerStarted","Data":"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984"} Sep 18 22:31:36.275000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:36.274960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" event={"ID":"5d4b1359-4942-4ace-8cae-9cb648e2cf6c","Type":"ContainerStarted","Data":"a1d98573bdb254eb08f747148f0038db1ce0e42b7430e2c204ec4c4c4a981023"} Sep 18 22:31:36.310263 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:36.310205 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" podStartSLOduration=2.310186134 podStartE2EDuration="2.310186134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:31:36.308299168 +0000 UTC m=+1881.979797138" watchObservedRunningTime="2026-09-18 22:31:36.310186134 +0000 UTC m=+1881.981684104" Sep 18 22:31:38.883200 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:38.883150 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:31:45.037530 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:45.037487 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:45.037530 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:45.037544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:45.050664 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:45.050631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:31:45.341676 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:31:45.341582 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:32:00.556346 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:00.556298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:32:02.007233 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:02.007193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:07.311531 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:07.311495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:07.633418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:07.633330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:07.792482 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:07.792448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:09.877647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:09.877609 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:10.548257 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:10.548223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:10.712845 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:10.712777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:11.403229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:11.403193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:13.121853 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:13.121779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:14.017284 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:14.017251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:14.867632 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:14.867598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:15.718842 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:15.718806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:15.819142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:15.819112 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:15.954620 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:15.954583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:16.413703 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.413668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:16.519216 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.519178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:16.619230 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.619189 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:16.624473 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.624449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.630535 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.630503 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3aa245-dockercfg-27wt5\"" Sep 18 22:32:16.630865 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.630848 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3aa245\"" Sep 18 22:32:16.639016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.638986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:16.665331 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.665242 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:16.669187 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.669165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.682227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.682191 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:16.691070 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.691036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.691247 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.691077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.691247 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.691111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rjb\" (UniqueName: \"kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.691247 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.691153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqb9c\" (UniqueName: \"kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.792501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.792454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.792723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.792518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rjb\" (UniqueName: \"kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.792723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.792580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqb9c\" (UniqueName: \"kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.792723 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.792719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.793323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.793294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.793588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.793565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.805869 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.805838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rjb\" (UniqueName: \"kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb\") pod \"payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:16.806019 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.805877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqb9c\" (UniqueName: \"kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c\") pod \"payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.936307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.936269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:16.980649 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:16.980598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:17.230903 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.230873 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:17.233540 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:32:17.233509 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f35e73a_0cb9_46c4_95f1_d4f2eabc2e1a.slice/crio-c460b1857506acb3c551f81690483b385db78898949f5dbab66daceff46c0ecc WatchSource:0}: Error finding container c460b1857506acb3c551f81690483b385db78898949f5dbab66daceff46c0ecc: Status 404 returned error can't find the container with id c460b1857506acb3c551f81690483b385db78898949f5dbab66daceff46c0ecc Sep 18 22:32:17.282480 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.282445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:17.288062 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:32:17.288022 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf159513d_eedd_4c08_af07_31e6b85dadd7.slice/crio-a7b24f89d63847c680628551455383cf1576c4366ac967d0ab5c2c08f5f1dc5e WatchSource:0}: Error finding container a7b24f89d63847c680628551455383cf1576c4366ac967d0ab5c2c08f5f1dc5e: Status 404 returned error can't find the container with id a7b24f89d63847c680628551455383cf1576c4366ac967d0ab5c2c08f5f1dc5e Sep 18 22:32:17.501534 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.501424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" event={"ID":"f159513d-eedd-4c08-af07-31e6b85dadd7","Type":"ContainerStarted","Data":"97e5eca06066ee12efc9648a8e81711dcc5bd000870d248d324f41d6205c0f89"} Sep 18 22:32:17.501534 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.501488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" event={"ID":"f159513d-eedd-4c08-af07-31e6b85dadd7","Type":"ContainerStarted","Data":"a7b24f89d63847c680628551455383cf1576c4366ac967d0ab5c2c08f5f1dc5e"} Sep 18 22:32:17.501783 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.501533 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:17.503386 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.503345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" event={"ID":"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a","Type":"ContainerStarted","Data":"cce73d68f489f954bf9dbf830388691cafb81153b3726644887b57116156de1e"} Sep 18 22:32:17.503536 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.503396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" event={"ID":"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a","Type":"ContainerStarted","Data":"c460b1857506acb3c551f81690483b385db78898949f5dbab66daceff46c0ecc"} Sep 18 22:32:17.503536 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.503420 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:17.527803 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:17.527732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" podStartSLOduration=1.527708016 podStartE2EDuration="1.527708016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:17.525815253 +0000 UTC m=+1923.197313224" watchObservedRunningTime="2026-09-18 22:32:17.527708016 +0000 UTC m=+1923.199205986" Sep 18 22:32:21.064206 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:21.064139 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" podStartSLOduration=5.064118567 podStartE2EDuration="5.064118567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:17.549411547 +0000 UTC m=+1923.220909531" watchObservedRunningTime="2026-09-18 22:32:21.064118567 +0000 UTC m=+1926.735616536" Sep 18 22:32:21.064721 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:21.064556 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:32:27.129679 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:27.129641 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:28.511512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:28.511477 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:28.511973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:28.511559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:36.955874 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:36.955837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:37.884048 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:37.883999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:39.021367 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:39.021332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:39.791376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:39.791343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:40.672844 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:40.672812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:40.811938 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:40.811905 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:41.934562 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:41.934524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:43.047554 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:43.047518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:44.004647 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:44.004615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:44.974980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:44.974939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:45.916269 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:45.916239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:46.754261 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:46.754233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:52.633987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:52.633943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.501956 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.501921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.557894 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.557858 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:53.558173 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.558145 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" podUID="f159513d-eedd-4c08-af07-31e6b85dadd7" containerName="payload-processing" containerID="cri-o://97e5eca06066ee12efc9648a8e81711dcc5bd000870d248d324f41d6205c0f89" gracePeriod=30 Sep 18 22:32:53.576046 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.576000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:53.576316 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.576281 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" podUID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" containerName="payload-pre-processing" containerID="cri-o://cce73d68f489f954bf9dbf830388691cafb81153b3726644887b57116156de1e" gracePeriod=30 Sep 18 22:32:53.634209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.634175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.703864 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.703814 2577 generic.go:356] "Generic (PLEG): container finished" podID="f159513d-eedd-4c08-af07-31e6b85dadd7" containerID="97e5eca06066ee12efc9648a8e81711dcc5bd000870d248d324f41d6205c0f89" exitCode=0 Sep 18 22:32:53.704055 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.703912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" event={"ID":"f159513d-eedd-4c08-af07-31e6b85dadd7","Type":"ContainerDied","Data":"97e5eca06066ee12efc9648a8e81711dcc5bd000870d248d324f41d6205c0f89"} Sep 18 22:32:53.705998 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.705964 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" containerID="cce73d68f489f954bf9dbf830388691cafb81153b3726644887b57116156de1e" exitCode=0 Sep 18 22:32:53.706157 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.706040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" event={"ID":"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a","Type":"ContainerDied","Data":"cce73d68f489f954bf9dbf830388691cafb81153b3726644887b57116156de1e"} Sep 18 22:32:53.732397 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.732362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.830566 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.830525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.865868 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.865840 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:53.869050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.869020 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:53.926637 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.926598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:53.967100 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume\") pod \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " Sep 18 22:32:53.967100 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqb9c\" (UniqueName: \"kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c\") pod \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\" (UID: \"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a\") " Sep 18 22:32:53.967369 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28rjb\" (UniqueName: \"kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb\") pod \"f159513d-eedd-4c08-af07-31e6b85dadd7\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " Sep 18 22:32:53.967369 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume\") pod \"f159513d-eedd-4c08-af07-31e6b85dadd7\" (UID: \"f159513d-eedd-4c08-af07-31e6b85dadd7\") " Sep 18 22:32:53.967528 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" (UID: "8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:32:53.967756 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.967732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f159513d-eedd-4c08-af07-31e6b85dadd7" (UID: "f159513d-eedd-4c08-af07-31e6b85dadd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:32:53.969645 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.969613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c" (OuterVolumeSpecName: "kube-api-access-mqb9c") pod "8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" (UID: "8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a"). InnerVolumeSpecName "kube-api-access-mqb9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:32:53.969645 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:53.969621 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb" (OuterVolumeSpecName: "kube-api-access-28rjb") pod "f159513d-eedd-4c08-af07-31e6b85dadd7" (UID: "f159513d-eedd-4c08-af07-31e6b85dadd7"). InnerVolumeSpecName "kube-api-access-28rjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:32:54.068971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.068874 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rjb\" (UniqueName: \"kubernetes.io/projected/f159513d-eedd-4c08-af07-31e6b85dadd7-kube-api-access-28rjb\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:32:54.068971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.068910 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f159513d-eedd-4c08-af07-31e6b85dadd7-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:32:54.068971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.068920 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:32:54.068971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.068928 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqb9c\" (UniqueName: \"kubernetes.io/projected/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a-kube-api-access-mqb9c\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:32:54.288743 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.288705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:54.390779 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.390687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:54.712363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.712325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" event={"ID":"f159513d-eedd-4c08-af07-31e6b85dadd7","Type":"ContainerDied","Data":"a7b24f89d63847c680628551455383cf1576c4366ac967d0ab5c2c08f5f1dc5e"} Sep 18 22:32:54.712363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.712361 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65" Sep 18 22:32:54.712989 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.712407 2577 scope.go:117] "RemoveContainer" containerID="97e5eca06066ee12efc9648a8e81711dcc5bd000870d248d324f41d6205c0f89" Sep 18 22:32:54.714100 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.714076 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" Sep 18 22:32:54.714190 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.714103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7" event={"ID":"8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a","Type":"ContainerDied","Data":"c460b1857506acb3c551f81690483b385db78898949f5dbab66daceff46c0ecc"} Sep 18 22:32:54.722399 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.722376 2577 scope.go:117] "RemoveContainer" containerID="cce73d68f489f954bf9dbf830388691cafb81153b3726644887b57116156de1e" Sep 18 22:32:54.747356 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.747318 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:54.751043 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.750983 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3aa245-59c7f7d78-bjsf7"] Sep 18 22:32:54.763083 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.763049 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:54.769836 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.769803 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3aa245-85bf5869c5-8vx65"] Sep 18 22:32:54.849904 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.849865 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" path="/var/lib/kubelet/pods/8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a/volumes" Sep 18 22:32:54.850307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:54.850291 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f159513d-eedd-4c08-af07-31e6b85dadd7" path="/var/lib/kubelet/pods/f159513d-eedd-4c08-af07-31e6b85dadd7/volumes" Sep 18 22:32:55.551712 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:55.551673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:56.609654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:56.609616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:57.510056 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:57.510021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:32:57.819947 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:57.819849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:32:58.689151 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:32:58.689120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:00.801485 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:00.801451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:00.903051 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:00.903014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:01.094340 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:01.094251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:05.026999 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:05.026962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:05.982444 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:05.982409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:07.172604 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:07.172569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:08.536610 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:08.536571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:09.467179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:09.467141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:10.344140 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:10.344107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:18.147720 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:18.147674 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:33:20.629215 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.629185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:20.664997 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.664961 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:33:20.665218 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.665191 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" podUID="ec4d6288-8437-4f22-b13d-59affaa5b64c" containerName="payload-pre-processing" containerID="cri-o://01ea5517546e318e3ed9ab9ef4e5e21c74e75626c7044be6a568247bd4f86e1f" gracePeriod=30 Sep 18 22:33:20.728719 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.728675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:20.829069 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.829031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:20.857823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.857765 2577 generic.go:356] "Generic (PLEG): container finished" podID="ec4d6288-8437-4f22-b13d-59affaa5b64c" containerID="01ea5517546e318e3ed9ab9ef4e5e21c74e75626c7044be6a568247bd4f86e1f" exitCode=0 Sep 18 22:33:20.858008 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.857825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" event={"ID":"ec4d6288-8437-4f22-b13d-59affaa5b64c","Type":"ContainerDied","Data":"01ea5517546e318e3ed9ab9ef4e5e21c74e75626c7044be6a568247bd4f86e1f"} Sep 18 22:33:20.939317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.939276 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:33:20.958224 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:20.958188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:21.036434 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.036392 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume\") pod \"ec4d6288-8437-4f22-b13d-59affaa5b64c\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " Sep 18 22:33:21.036640 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.036466 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd9v7\" (UniqueName: \"kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7\") pod \"ec4d6288-8437-4f22-b13d-59affaa5b64c\" (UID: \"ec4d6288-8437-4f22-b13d-59affaa5b64c\") " Sep 18 22:33:21.036971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.036942 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec4d6288-8437-4f22-b13d-59affaa5b64c" (UID: "ec4d6288-8437-4f22-b13d-59affaa5b64c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:33:21.038953 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.038922 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7" (OuterVolumeSpecName: "kube-api-access-cd9v7") pod "ec4d6288-8437-4f22-b13d-59affaa5b64c" (UID: "ec4d6288-8437-4f22-b13d-59affaa5b64c"). InnerVolumeSpecName "kube-api-access-cd9v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:33:21.137344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.137303 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4d6288-8437-4f22-b13d-59affaa5b64c-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:21.137344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.137340 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd9v7\" (UniqueName: \"kubernetes.io/projected/ec4d6288-8437-4f22-b13d-59affaa5b64c-kube-api-access-cd9v7\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:21.327987 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.327942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:21.437160 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.437116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:21.864016 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.863979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" event={"ID":"ec4d6288-8437-4f22-b13d-59affaa5b64c","Type":"ContainerDied","Data":"cacfdd840be39a5c53a884edb04e0252a8a795639114bd44ac803f4f22b6424a"} Sep 18 22:33:21.864439 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.864024 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf" Sep 18 22:33:21.864439 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.864048 2577 scope.go:117] "RemoveContainer" containerID="01ea5517546e318e3ed9ab9ef4e5e21c74e75626c7044be6a568247bd4f86e1f" Sep 18 22:33:21.909086 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.909048 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:33:21.926741 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:21.926705 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-adba81-5ffd65b88b-g94sf"] Sep 18 22:33:22.667147 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:22.667107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:22.848567 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:22.848529 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4d6288-8437-4f22-b13d-59affaa5b64c" path="/var/lib/kubelet/pods/ec4d6288-8437-4f22-b13d-59affaa5b64c/volumes" Sep 18 22:33:23.721916 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:23.721877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:24.614894 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:24.614860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:26.305308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:26.305265 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:33:26.395543 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:26.395512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:31.243076 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:31.243034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:33:35.889524 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:35.889481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:33:41.325817 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.325751 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" podUID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" containerName="main" containerID="cri-o://b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984" gracePeriod=45 Sep 18 22:33:41.590313 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.590238 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:33:41.652062 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652032 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx8bg\" (UniqueName: \"kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652192 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652208 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm\") pod \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\" (UID: \"5d4b1359-4942-4ace-8cae-9cb648e2cf6c\") " Sep 18 22:33:41.652458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652302 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home" (OuterVolumeSpecName: "home") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:33:41.652560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:33:41.652560 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652528 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:33:41.652637 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.652539 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.654588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.654552 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm" (OuterVolumeSpecName: "dshm") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:33:41.654717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.654593 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:33:41.654717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.654623 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg" (OuterVolumeSpecName: "kube-api-access-zx8bg") pod "5d4b1359-4942-4ace-8cae-9cb648e2cf6c" (UID: "5d4b1359-4942-4ace-8cae-9cb648e2cf6c"). InnerVolumeSpecName "kube-api-access-zx8bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:33:41.753285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.753238 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.753285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.753274 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.753285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.753283 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.753285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.753291 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.753285 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.753299 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8bg\" (UniqueName: \"kubernetes.io/projected/5d4b1359-4942-4ace-8cae-9cb648e2cf6c-kube-api-access-zx8bg\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:33:41.978227 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.978196 2577 generic.go:356] "Generic (PLEG): container finished" podID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" containerID="b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984" exitCode=0 Sep 18 22:33:41.978417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.978286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" event={"ID":"5d4b1359-4942-4ace-8cae-9cb648e2cf6c","Type":"ContainerDied","Data":"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984"} Sep 18 22:33:41.978417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.978301 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" Sep 18 22:33:41.978417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.978315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml" event={"ID":"5d4b1359-4942-4ace-8cae-9cb648e2cf6c","Type":"ContainerDied","Data":"a1d98573bdb254eb08f747148f0038db1ce0e42b7430e2c204ec4c4c4a981023"} Sep 18 22:33:41.978417 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.978331 2577 scope.go:117] "RemoveContainer" containerID="b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984" Sep 18 22:33:41.987980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.987951 2577 scope.go:117] "RemoveContainer" containerID="b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984" Sep 18 22:33:41.988324 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:33:41.988302 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984\": container with ID starting with b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984 not found: ID does not exist" containerID="b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984" Sep 18 22:33:41.988380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:41.988334 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984"} err="failed to get container status \"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984\": rpc error: code = NotFound desc = could not find container \"b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984\": container with ID starting with b5bf927bf19127c63900c5639fedcafce641609a147e96cc2e7dab65baf1f984 not found: ID does not exist" Sep 18 22:33:45.439461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:45.439412 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:33:46.187568 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:46.187526 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-adba81/ipp-route-adba81-kserve-69598bf76c-49qml"] Sep 18 22:33:46.849080 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:33:46.849042 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" path="/var/lib/kubelet/pods/5d4b1359-4942-4ace-8cae-9cb648e2cf6c/volumes" Sep 18 22:34:03.494277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:03.494242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:04.655584 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:04.655545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:10.543049 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:10.543013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:10.649685 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:10.649635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:10.844070 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:10.843954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:11.695047 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:11.695014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:12.940774 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:12.940737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:13.752050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:13.752012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:14.563225 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:14.563189 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:15.659277 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:15.659243 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:21.815945 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.815906 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:34:21.816436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.816188 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" podUID="baa63048-1317-41f1-8a55-cd6b514f9218" containerName="maas-api" containerID="cri-o://1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76" gracePeriod=30 Sep 18 22:34:21.873329 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.873293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:21.919013 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.918974 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:34:21.919237 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.919209 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" podUID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" containerName="payload-pre-processing" containerID="cri-o://afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a" gracePeriod=30 Sep 18 22:34:21.994532 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:21.994503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:22.089657 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.089623 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:34:22.120407 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.120355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:22.149255 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.149223 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca\") pod \"baa63048-1317-41f1-8a55-cd6b514f9218\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " Sep 18 22:34:22.149458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.149297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") pod \"baa63048-1317-41f1-8a55-cd6b514f9218\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " Sep 18 22:34:22.149458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.149331 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs\") pod \"baa63048-1317-41f1-8a55-cd6b514f9218\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " Sep 18 22:34:22.149458 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.149387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx4qv\" (UniqueName: \"kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv\") pod \"baa63048-1317-41f1-8a55-cd6b514f9218\" (UID: \"baa63048-1317-41f1-8a55-cd6b514f9218\") " Sep 18 22:34:22.149703 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.149678 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "baa63048-1317-41f1-8a55-cd6b514f9218" (UID: "baa63048-1317-41f1-8a55-cd6b514f9218"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:34:22.152434 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.152398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "baa63048-1317-41f1-8a55-cd6b514f9218" (UID: "baa63048-1317-41f1-8a55-cd6b514f9218"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:34:22.152671 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.152648 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv" (OuterVolumeSpecName: "kube-api-access-lx4qv") pod "baa63048-1317-41f1-8a55-cd6b514f9218" (UID: "baa63048-1317-41f1-8a55-cd6b514f9218"). InnerVolumeSpecName "kube-api-access-lx4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:34:22.152812 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.152761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "baa63048-1317-41f1-8a55-cd6b514f9218" (UID: "baa63048-1317-41f1-8a55-cd6b514f9218"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:34:22.173929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.173902 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:34:22.185424 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.185395 2577 generic.go:356] "Generic (PLEG): container finished" podID="baa63048-1317-41f1-8a55-cd6b514f9218" containerID="1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76" exitCode=0 Sep 18 22:34:22.185615 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.185438 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" Sep 18 22:34:22.185615 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.185476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" event={"ID":"baa63048-1317-41f1-8a55-cd6b514f9218","Type":"ContainerDied","Data":"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76"} Sep 18 22:34:22.185615 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.185537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z" event={"ID":"baa63048-1317-41f1-8a55-cd6b514f9218","Type":"ContainerDied","Data":"ac67db889e73aa58b48917760a16175a82b196bb3a4d95ffd52f2f66642aef28"} Sep 18 22:34:22.185615 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.185571 2577 scope.go:117] "RemoveContainer" containerID="1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76" Sep 18 22:34:22.187088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.187063 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" containerID="afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a" exitCode=0 Sep 18 22:34:22.187179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.187097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" Sep 18 22:34:22.187179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.187111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" event={"ID":"c7b061bc-1cc8-4218-8bb0-ace8c951d633","Type":"ContainerDied","Data":"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a"} Sep 18 22:34:22.187179 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.187159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47" event={"ID":"c7b061bc-1cc8-4218-8bb0-ace8c951d633","Type":"ContainerDied","Data":"cf13260cb817e0cc3fcd2c8a087170f8b8af3741986f23ae496cb7198668b3b0"} Sep 18 22:34:22.195336 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.195317 2577 scope.go:117] "RemoveContainer" containerID="1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76" Sep 18 22:34:22.195695 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:34:22.195676 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76\": container with ID starting with 1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76 not found: ID does not exist" containerID="1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76" Sep 18 22:34:22.195750 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.195708 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76"} err="failed to get container status \"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76\": rpc error: code = NotFound desc = could not find container \"1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76\": container with ID starting with 1b3e9ac3913d8186fe7cc268e93380bc0206a8a41012155d89c7af7306a62e76 not found: ID does not exist" Sep 18 22:34:22.195750 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.195727 2577 scope.go:117] "RemoveContainer" containerID="afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a" Sep 18 22:34:22.204520 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.204499 2577 scope.go:117] "RemoveContainer" containerID="afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a" Sep 18 22:34:22.204900 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:34:22.204876 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a\": container with ID starting with afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a not found: ID does not exist" containerID="afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a" Sep 18 22:34:22.204967 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.204908 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a"} err="failed to get container status \"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a\": rpc error: code = NotFound desc = could not find container \"afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a\": container with ID starting with afb51a40e3cc67a74503082f7ecd3a435e0120cc47bacbc28cf0f68f6fde8e8a not found: ID does not exist" Sep 18 22:34:22.250775 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.250746 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume\") pod \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " Sep 18 22:34:22.250961 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.250854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjnjs\" (UniqueName: \"kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs\") pod \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\" (UID: \"c7b061bc-1cc8-4218-8bb0-ace8c951d633\") " Sep 18 22:34:22.251050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.251039 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.251091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.251052 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa63048-1317-41f1-8a55-cd6b514f9218-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.251091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.251061 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4qv\" (UniqueName: \"kubernetes.io/projected/baa63048-1317-41f1-8a55-cd6b514f9218-kube-api-access-lx4qv\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.251091 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.251069 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa63048-1317-41f1-8a55-cd6b514f9218-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.251221 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.251194 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7b061bc-1cc8-4218-8bb0-ace8c951d633" (UID: "c7b061bc-1cc8-4218-8bb0-ace8c951d633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:34:22.253180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.253152 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs" (OuterVolumeSpecName: "kube-api-access-vjnjs") pod "c7b061bc-1cc8-4218-8bb0-ace8c951d633" (UID: "c7b061bc-1cc8-4218-8bb0-ace8c951d633"). InnerVolumeSpecName "kube-api-access-vjnjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:34:22.270634 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.270595 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:34:22.279830 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.279767 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-58a8ca-674696d558-bqk5z"] Sep 18 22:34:22.294555 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.294520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:22.351739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.351650 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjnjs\" (UniqueName: \"kubernetes.io/projected/c7b061bc-1cc8-4218-8bb0-ace8c951d633-kube-api-access-vjnjs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.351739 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.351680 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b061bc-1cc8-4218-8bb0-ace8c951d633-plugins-config-volume\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:22.518955 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.518918 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:34:22.528036 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.528004 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-58a8ca-597c4d899d-tck47"] Sep 18 22:34:22.552364 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.552333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:22.760759 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.760718 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:22.848670 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.848632 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa63048-1317-41f1-8a55-cd6b514f9218" path="/var/lib/kubelet/pods/baa63048-1317-41f1-8a55-cd6b514f9218/volumes" Sep 18 22:34:22.849088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.849072 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" path="/var/lib/kubelet/pods/c7b061bc-1cc8-4218-8bb0-ace8c951d633/volumes" Sep 18 22:34:22.974996 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:22.974964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:23.940920 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:23.940877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:24.884802 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:24.884750 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:25.320154 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:25.320114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:34:29.122049 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:29.122018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:29.222291 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:29.222257 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:29.403708 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:29.403624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:30.180721 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:30.180678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:31.143284 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:31.143242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:32.304879 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:32.304846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:33.238427 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:33.238393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:39.860728 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:39.860642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:40.402605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.402571 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:34:40.402888 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.402842 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" podUID="4b670e63-24b4-41d7-b727-069a41d0d83d" containerName="maas-api" containerID="cri-o://7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b" gracePeriod=30 Sep 18 22:34:40.439078 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.439035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:40.570383 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.570352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:40.659530 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.659458 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:34:40.723448 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.723409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:40.825911 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.825874 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs\") pod \"4b670e63-24b4-41d7-b727-069a41d0d83d\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " Sep 18 22:34:40.826090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.825943 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca\") pod \"4b670e63-24b4-41d7-b727-069a41d0d83d\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " Sep 18 22:34:40.826090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.825979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls\") pod \"4b670e63-24b4-41d7-b727-069a41d0d83d\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " Sep 18 22:34:40.826090 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.826033 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq\") pod \"4b670e63-24b4-41d7-b727-069a41d0d83d\" (UID: \"4b670e63-24b4-41d7-b727-069a41d0d83d\") " Sep 18 22:34:40.826446 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.826416 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b670e63-24b4-41d7-b727-069a41d0d83d" (UID: "4b670e63-24b4-41d7-b727-069a41d0d83d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:34:40.828266 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.828228 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b670e63-24b4-41d7-b727-069a41d0d83d" (UID: "4b670e63-24b4-41d7-b727-069a41d0d83d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:34:40.828385 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.828287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b670e63-24b4-41d7-b727-069a41d0d83d" (UID: "4b670e63-24b4-41d7-b727-069a41d0d83d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:34:40.828385 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.828338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq" (OuterVolumeSpecName: "kube-api-access-vq5wq") pod "4b670e63-24b4-41d7-b727-069a41d0d83d" (UID: "4b670e63-24b4-41d7-b727-069a41d0d83d"). InnerVolumeSpecName "kube-api-access-vq5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:34:40.867145 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.867111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:40.927418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.927318 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:40.927418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.927353 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b670e63-24b4-41d7-b727-069a41d0d83d-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:40.927418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.927366 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b670e63-24b4-41d7-b727-069a41d0d83d-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:40.927418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.927378 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5wq\" (UniqueName: \"kubernetes.io/projected/4b670e63-24b4-41d7-b727-069a41d0d83d-kube-api-access-vq5wq\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:34:40.954628 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:40.954595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:41.299376 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.299338 2577 generic.go:356] "Generic (PLEG): container finished" podID="4b670e63-24b4-41d7-b727-069a41d0d83d" containerID="7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b" exitCode=0 Sep 18 22:34:41.299591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.299381 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" Sep 18 22:34:41.299591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.299420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" event={"ID":"4b670e63-24b4-41d7-b727-069a41d0d83d","Type":"ContainerDied","Data":"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b"} Sep 18 22:34:41.299591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.299475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk" event={"ID":"4b670e63-24b4-41d7-b727-069a41d0d83d","Type":"ContainerDied","Data":"743915c2d0afd9589355a53137daa6201e71d26316436a6a7521402c76db1ee3"} Sep 18 22:34:41.299591 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.299498 2577 scope.go:117] "RemoveContainer" containerID="7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b" Sep 18 22:34:41.309159 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.309137 2577 scope.go:117] "RemoveContainer" containerID="7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b" Sep 18 22:34:41.309509 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:34:41.309488 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b\": container with ID starting with 7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b not found: ID does not exist" containerID="7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b" Sep 18 22:34:41.309551 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.309522 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b"} err="failed to get container status \"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b\": rpc error: code = NotFound desc = could not find container \"7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b\": container with ID starting with 7ca10c7f1b2eebe504a51b8349b311a36afa065df6a76cc01bd26a0fedcf3c9b not found: ID does not exist" Sep 18 22:34:41.334247 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.334212 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:34:41.346577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.346540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:41.346902 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.346882 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-123a30-6868db9cf8-xswvk"] Sep 18 22:34:41.537288 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:41.537256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:42.374685 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:42.374654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:42.603668 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:42.603633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:42.849035 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:42.848999 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b670e63-24b4-41d7-b727-069a41d0d83d" path="/var/lib/kubelet/pods/4b670e63-24b4-41d7-b727-069a41d0d83d/volumes" Sep 18 22:34:43.027337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:43.027309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:43.669335 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:43.669298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:34:48.869348 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:48.869310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:48.969607 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:48.969572 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:49.157350 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:49.157260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:49.653488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:49.653431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:50.743777 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:50.743738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:34:51.746209 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:34:51.746161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:00.295114 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.295068 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:35:00.295582 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.295308 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" podUID="79ab7647-fa08-419a-8d3b-58dadb46d012" containerName="maas-api" containerID="cri-o://f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927" gracePeriod=30 Sep 18 22:35:00.428229 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.428196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:00.560330 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.560256 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:35:00.616717 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.616678 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca\") pod \"79ab7647-fa08-419a-8d3b-58dadb46d012\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " Sep 18 22:35:00.616923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.616827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdpd\" (UniqueName: \"kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd\") pod \"79ab7647-fa08-419a-8d3b-58dadb46d012\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " Sep 18 22:35:00.616923 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.616910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs\") pod \"79ab7647-fa08-419a-8d3b-58dadb46d012\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " Sep 18 22:35:00.617033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.617012 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") pod \"79ab7647-fa08-419a-8d3b-58dadb46d012\" (UID: \"79ab7647-fa08-419a-8d3b-58dadb46d012\") " Sep 18 22:35:00.617140 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.617115 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79ab7647-fa08-419a-8d3b-58dadb46d012" (UID: "79ab7647-fa08-419a-8d3b-58dadb46d012"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:35:00.617307 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.617290 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab7647-fa08-419a-8d3b-58dadb46d012-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:00.619185 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.619155 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79ab7647-fa08-419a-8d3b-58dadb46d012" (UID: "79ab7647-fa08-419a-8d3b-58dadb46d012"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:35:00.619276 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.619246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79ab7647-fa08-419a-8d3b-58dadb46d012" (UID: "79ab7647-fa08-419a-8d3b-58dadb46d012"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:35:00.619276 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.619245 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd" (OuterVolumeSpecName: "kube-api-access-jzdpd") pod "79ab7647-fa08-419a-8d3b-58dadb46d012" (UID: "79ab7647-fa08-419a-8d3b-58dadb46d012"). InnerVolumeSpecName "kube-api-access-jzdpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:35:00.647317 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.647277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:00.718489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.718443 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:00.718489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.718484 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab7647-fa08-419a-8d3b-58dadb46d012-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:00.718489 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.718495 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdpd\" (UniqueName: \"kubernetes.io/projected/79ab7647-fa08-419a-8d3b-58dadb46d012-kube-api-access-jzdpd\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:00.795308 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.795266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:00.872182 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:00.872152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:01.215418 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.215387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:01.314085 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.314048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:01.400084 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.400049 2577 generic.go:356] "Generic (PLEG): container finished" podID="79ab7647-fa08-419a-8d3b-58dadb46d012" containerID="f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927" exitCode=0 Sep 18 22:35:01.400292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.400097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" Sep 18 22:35:01.400292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.400140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" event={"ID":"79ab7647-fa08-419a-8d3b-58dadb46d012","Type":"ContainerDied","Data":"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927"} Sep 18 22:35:01.400292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.400201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx" event={"ID":"79ab7647-fa08-419a-8d3b-58dadb46d012","Type":"ContainerDied","Data":"13ce4cab53bdc6ea620cca4d254b0db2c6e407364f110c82da214f2cc7e30f83"} Sep 18 22:35:01.400292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.400232 2577 scope.go:117] "RemoveContainer" containerID="f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927" Sep 18 22:35:01.411557 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.411530 2577 scope.go:117] "RemoveContainer" containerID="f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927" Sep 18 22:35:01.412025 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:35:01.412002 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927\": container with ID starting with f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927 not found: ID does not exist" containerID="f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927" Sep 18 22:35:01.412103 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.412033 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927"} err="failed to get container status \"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927\": rpc error: code = NotFound desc = could not find container \"f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927\": container with ID starting with f52e23a0710de30e5d27f1f7321c89351f921449800102466a49371962a1a927 not found: ID does not exist" Sep 18 22:35:01.424526 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.424480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:35:01.428459 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:01.428424 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7f10d8-59c87cd585-jfbqx"] Sep 18 22:35:02.848815 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:02.848753 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ab7647-fa08-419a-8d3b-58dadb46d012" path="/var/lib/kubelet/pods/79ab7647-fa08-419a-8d3b-58dadb46d012/volumes" Sep 18 22:35:03.216958 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:03.216914 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:35:08.875762 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:08.875730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:09.235180 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:09.235139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:09.522497 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:09.522404 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:12.208513 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.208463 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:35:12.209050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.208771 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b57d79649-fdn2s" podUID="68522970-3c26-4ead-aa07-0b3f14f9577f" containerName="manager" containerID="cri-o://6f8022de28d6f016859c0d0fbd295afde6e7d9a41b69d88a227a773676afdf34" gracePeriod=30 Sep 18 22:35:12.456557 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.456522 2577 generic.go:356] "Generic (PLEG): container finished" podID="68522970-3c26-4ead-aa07-0b3f14f9577f" containerID="6f8022de28d6f016859c0d0fbd295afde6e7d9a41b69d88a227a773676afdf34" exitCode=0 Sep 18 22:35:12.456725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.456596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d79649-fdn2s" event={"ID":"68522970-3c26-4ead-aa07-0b3f14f9577f","Type":"ContainerDied","Data":"6f8022de28d6f016859c0d0fbd295afde6e7d9a41b69d88a227a773676afdf34"} Sep 18 22:35:12.456725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.456651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d79649-fdn2s" event={"ID":"68522970-3c26-4ead-aa07-0b3f14f9577f","Type":"ContainerDied","Data":"1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8"} Sep 18 22:35:12.456725 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.456673 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4a5492c59bf95afdf1c4be53b81e0644ab24b7f1c2e90a28476792ed8546f8" Sep 18 22:35:12.466581 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.466508 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:35:12.630540 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.630501 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs\") pod \"68522970-3c26-4ead-aa07-0b3f14f9577f\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " Sep 18 22:35:12.630861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.630567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert\") pod \"68522970-3c26-4ead-aa07-0b3f14f9577f\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " Sep 18 22:35:12.630861 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.630620 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtz8g\" (UniqueName: \"kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g\") pod \"68522970-3c26-4ead-aa07-0b3f14f9577f\" (UID: \"68522970-3c26-4ead-aa07-0b3f14f9577f\") " Sep 18 22:35:12.633030 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.632982 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68522970-3c26-4ead-aa07-0b3f14f9577f" (UID: "68522970-3c26-4ead-aa07-0b3f14f9577f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:35:12.633030 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.632993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68522970-3c26-4ead-aa07-0b3f14f9577f" (UID: "68522970-3c26-4ead-aa07-0b3f14f9577f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:35:12.633197 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.633115 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g" (OuterVolumeSpecName: "kube-api-access-xtz8g") pod "68522970-3c26-4ead-aa07-0b3f14f9577f" (UID: "68522970-3c26-4ead-aa07-0b3f14f9577f"). InnerVolumeSpecName "kube-api-access-xtz8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:35:12.732322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.732224 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:12.732322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.732258 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68522970-3c26-4ead-aa07-0b3f14f9577f-webhook-cert\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:12.732322 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:12.732268 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtz8g\" (UniqueName: \"kubernetes.io/projected/68522970-3c26-4ead-aa07-0b3f14f9577f-kube-api-access-xtz8g\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:35:13.433363 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:13.433321 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:35:13.461292 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:13.461250 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d79649-fdn2s" Sep 18 22:35:13.488880 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:13.488837 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:35:13.497530 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:13.497486 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b57d79649-fdn2s"] Sep 18 22:35:14.848858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:14.848823 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68522970-3c26-4ead-aa07-0b3f14f9577f" path="/var/lib/kubelet/pods/68522970-3c26-4ead-aa07-0b3f14f9577f/volumes" Sep 18 22:35:15.146974 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:15.146890 2577 scope.go:117] "RemoveContainer" containerID="6f8022de28d6f016859c0d0fbd295afde6e7d9a41b69d88a227a773676afdf34" Sep 18 22:35:42.294042 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:42.294001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:42.359088 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:42.359046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:35:51.986344 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:35:51.986309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:36:35.978571 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:36:35.978536 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:36:36.216332 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:36:36.216276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:36:37.548883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:36:37.548846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:37:15.317306 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:37:15.317256 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:37:23.208401 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:37:23.208361 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:37:31.416388 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:37:31.416340 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:37:49.435368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:37:49.435270 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:37:56.110748 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:37:56.110704 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:38:04.086081 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:38:04.086042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:39:31.836368 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:31.836321 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:39:37.836263 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:37.836218 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:39:44.927654 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.927615 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928029 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b670e63-24b4-41d7-b727-069a41d0d83d" containerName="maas-api" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928044 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b670e63-24b4-41d7-b727-069a41d0d83d" containerName="maas-api" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928057 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f159513d-eedd-4c08-af07-31e6b85dadd7" containerName="payload-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928065 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f159513d-eedd-4c08-af07-31e6b85dadd7" containerName="payload-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928074 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" containerName="main" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" containerName="main" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928094 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" containerName="payload-pre-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928099 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" containerName="payload-pre-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928105 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68522970-3c26-4ead-aa07-0b3f14f9577f" containerName="manager" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928112 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="68522970-3c26-4ead-aa07-0b3f14f9577f" containerName="manager" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928121 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa63048-1317-41f1-8a55-cd6b514f9218" containerName="maas-api" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928126 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa63048-1317-41f1-8a55-cd6b514f9218" containerName="maas-api" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928133 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" containerName="payload-pre-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928143 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" containerName="payload-pre-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928154 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4d6288-8437-4f22-b13d-59affaa5b64c" containerName="payload-pre-processing" Sep 18 22:39:44.928156 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928160 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4d6288-8437-4f22-b13d-59affaa5b64c" containerName="payload-pre-processing" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928170 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ab7647-fa08-419a-8d3b-58dadb46d012" containerName="maas-api" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928175 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ab7647-fa08-419a-8d3b-58dadb46d012" containerName="maas-api" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928232 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4d6288-8437-4f22-b13d-59affaa5b64c" containerName="payload-pre-processing" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928240 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ab7647-fa08-419a-8d3b-58dadb46d012" containerName="maas-api" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928245 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa63048-1317-41f1-8a55-cd6b514f9218" containerName="maas-api" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928252 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="68522970-3c26-4ead-aa07-0b3f14f9577f" containerName="manager" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928258 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b061bc-1cc8-4218-8bb0-ace8c951d633" containerName="payload-pre-processing" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928264 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b670e63-24b4-41d7-b727-069a41d0d83d" containerName="maas-api" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928269 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f35e73a-0cb9-46c4-95f1-d4f2eabc2e1a" containerName="payload-pre-processing" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928274 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4b1359-4942-4ace-8cae-9cb648e2cf6c" containerName="main" Sep 18 22:39:44.928689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.928278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f159513d-eedd-4c08-af07-31e6b85dadd7" containerName="payload-processing" Sep 18 22:39:44.931448 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.931427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:44.935116 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.935089 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-361c0305-kserve-self-signed-certs\"" Sep 18 22:39:44.950922 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:44.950890 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:39:45.026578 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.026578 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whc4f\" (UniqueName: \"kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.026918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.026918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.026918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.026918 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.026842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.046735 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.046697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:39:45.127671 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.127671 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.127984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.127984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.127984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whc4f\" (UniqueName: \"kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.127984 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.127864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.128201 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.128161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.128286 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.128266 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.128335 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.128285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.130883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.130856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.131027 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.131011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.137004 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.136980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whc4f\" (UniqueName: \"kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f\") pod \"e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.243466 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.243431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:45.519545 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.519516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:39:45.521831 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:39:45.521775 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b045a6_7612_4f13_93de_d12640f27750.slice/crio-a37d680f6b39785250d99021c94b01090cbe6a77b558689945215bc32939db9e WatchSource:0}: Error finding container a37d680f6b39785250d99021c94b01090cbe6a77b558689945215bc32939db9e: Status 404 returned error can't find the container with id a37d680f6b39785250d99021c94b01090cbe6a77b558689945215bc32939db9e Sep 18 22:39:45.779516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.779418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" event={"ID":"36b045a6-7612-4f13-93de-d12640f27750","Type":"ContainerStarted","Data":"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7"} Sep 18 22:39:45.779516 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.779456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" event={"ID":"36b045a6-7612-4f13-93de-d12640f27750","Type":"ContainerStarted","Data":"a37d680f6b39785250d99021c94b01090cbe6a77b558689945215bc32939db9e"} Sep 18 22:39:45.800716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:45.800663 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" podStartSLOduration=1.800644527 podStartE2EDuration="1.800644527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:39:45.797851498 +0000 UTC m=+2371.469349469" watchObservedRunningTime="2026-09-18 22:39:45.800644527 +0000 UTC m=+2371.472142497" Sep 18 22:39:47.631823 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:47.631768 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:39:55.243908 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:55.243873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:55.244338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:55.243928 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:55.258373 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:55.258328 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:39:55.839686 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:39:55.839652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:40:03.524426 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:03.524393 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:40:06.537380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:06.537340 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:18.411354 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:18.411313 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:40:18.437765 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:18.437734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:40:20.596816 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:20.595308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:21.122028 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:21.121993 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:40:28.527474 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:28.527432 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:33.432716 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.432672 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" podUID="36b045a6-7612-4f13-93de-d12640f27750" containerName="main" containerID="cri-o://c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7" gracePeriod=45 Sep 18 22:40:33.683947 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.683863 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:40:33.797189 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whc4f\" (UniqueName: \"kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797212 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797269 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797295 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797380 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797325 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797580 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797392 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir\") pod \"36b045a6-7612-4f13-93de-d12640f27750\" (UID: \"36b045a6-7612-4f13-93de-d12640f27750\") " Sep 18 22:40:33.797580 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache" (OuterVolumeSpecName: "model-cache") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:40:33.797694 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797587 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home" (OuterVolumeSpecName: "home") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:40:33.797751 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797700 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:40:33.797751 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797735 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-home\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:33.797859 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.797754 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-model-cache\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:33.799880 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.799850 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f" (OuterVolumeSpecName: "kube-api-access-whc4f") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "kube-api-access-whc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:40:33.799880 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.799859 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:40:33.800067 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.799907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm" (OuterVolumeSpecName: "dshm") pod "36b045a6-7612-4f13-93de-d12640f27750" (UID: "36b045a6-7612-4f13-93de-d12640f27750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 22:40:33.898488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.898447 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whc4f\" (UniqueName: \"kubernetes.io/projected/36b045a6-7612-4f13-93de-d12640f27750-kube-api-access-whc4f\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:33.898488 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.898488 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-dshm\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:33.898780 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.898502 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b045a6-7612-4f13-93de-d12640f27750-tls-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:33.898780 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:33.898514 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b045a6-7612-4f13-93de-d12640f27750-tmp-dir\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:34.014262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.014171 2577 generic.go:356] "Generic (PLEG): container finished" podID="36b045a6-7612-4f13-93de-d12640f27750" containerID="c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7" exitCode=0 Sep 18 22:40:34.014262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.014218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" Sep 18 22:40:34.014262 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.014236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" event={"ID":"36b045a6-7612-4f13-93de-d12640f27750","Type":"ContainerDied","Data":"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7"} Sep 18 22:40:34.014523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.014280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r" event={"ID":"36b045a6-7612-4f13-93de-d12640f27750","Type":"ContainerDied","Data":"a37d680f6b39785250d99021c94b01090cbe6a77b558689945215bc32939db9e"} Sep 18 22:40:34.014523 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.014303 2577 scope.go:117] "RemoveContainer" containerID="c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7" Sep 18 22:40:34.026511 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.026468 2577 scope.go:117] "RemoveContainer" containerID="c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7" Sep 18 22:40:34.027047 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:40:34.027016 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7\": container with ID starting with c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7 not found: ID does not exist" containerID="c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7" Sep 18 22:40:34.027129 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.027052 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7"} err="failed to get container status \"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7\": rpc error: code = NotFound desc = could not find container \"c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7\": container with ID starting with c80a22cd812fe1fba771933faaa08c660e9078c78c150ba0ea5b0b12de182dd7 not found: ID does not exist" Sep 18 22:40:34.038102 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.038065 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:40:34.040633 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.040602 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-361c0305-kserve-64fcd4845f-tpt6r"] Sep 18 22:40:34.696337 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.696299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:40:34.771323 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.771294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:40:34.849109 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:34.849070 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b045a6-7612-4f13-93de-d12640f27750" path="/var/lib/kubelet/pods/36b045a6-7612-4f13-93de-d12640f27750/volumes" Sep 18 22:40:37.711096 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.710999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:37.957150 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.957113 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9"] Sep 18 22:40:37.957506 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.957494 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b045a6-7612-4f13-93de-d12640f27750" containerName="main" Sep 18 22:40:37.957552 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.957508 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b045a6-7612-4f13-93de-d12640f27750" containerName="main" Sep 18 22:40:37.957593 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.957578 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b045a6-7612-4f13-93de-d12640f27750" containerName="main" Sep 18 22:40:37.962065 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.962013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:37.972841 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:37.972808 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9"] Sep 18 22:40:38.037561 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.037522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-metrics-certs\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.037758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.037576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87phn\" (UniqueName: \"kubernetes.io/projected/88fb8fef-9ee8-472a-b844-37eb52ccb579-kube-api-access-87phn\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.037758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.037643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fb8fef-9ee8-472a-b844-37eb52ccb579-openshift-service-ca\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.037758 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.037678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-maas-api-tls\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.138755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.138717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fb8fef-9ee8-472a-b844-37eb52ccb579-openshift-service-ca\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.138755 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.138760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-maas-api-tls\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.139021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.138865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-metrics-certs\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.139021 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.138916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87phn\" (UniqueName: \"kubernetes.io/projected/88fb8fef-9ee8-472a-b844-37eb52ccb579-kube-api-access-87phn\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.139565 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.139539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fb8fef-9ee8-472a-b844-37eb52ccb579-openshift-service-ca\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.141577 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.141554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-maas-api-tls\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.141695 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.141623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88fb8fef-9ee8-472a-b844-37eb52ccb579-metrics-certs\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.150338 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.150305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87phn\" (UniqueName: \"kubernetes.io/projected/88fb8fef-9ee8-472a-b844-37eb52ccb579-kube-api-access-87phn\") pod \"maas-api-6b86cc866f-r9nd9\" (UID: \"88fb8fef-9ee8-472a-b844-37eb52ccb579\") " pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.273138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.272997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:38.557761 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.557729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9"] Sep 18 22:40:38.560309 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:40:38.560279 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fb8fef_9ee8_472a_b844_37eb52ccb579.slice/crio-7b3e123b0e266baa7558f4f4d5a7dd2816f5693a16cd8eef49f3e0cfb5bd89d5 WatchSource:0}: Error finding container 7b3e123b0e266baa7558f4f4d5a7dd2816f5693a16cd8eef49f3e0cfb5bd89d5: Status 404 returned error can't find the container with id 7b3e123b0e266baa7558f4f4d5a7dd2816f5693a16cd8eef49f3e0cfb5bd89d5 Sep 18 22:40:38.561819 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:38.561776 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 22:40:39.042501 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:39.042467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" event={"ID":"88fb8fef-9ee8-472a-b844-37eb52ccb579","Type":"ContainerStarted","Data":"a45dacc76cc8ffcf0d3293f51e5e3ca22e2ef2f1aa666d7dc4d315e9f2ba5f8c"} Sep 18 22:40:39.042929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:39.042530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" event={"ID":"88fb8fef-9ee8-472a-b844-37eb52ccb579","Type":"ContainerStarted","Data":"7b3e123b0e266baa7558f4f4d5a7dd2816f5693a16cd8eef49f3e0cfb5bd89d5"} Sep 18 22:40:39.042929 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:39.042581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:39.063569 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:39.063502 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" podStartSLOduration=1.6731124309999998 podStartE2EDuration="2.063477014s" totalImagesPullingTime="390.364583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:40:39.062547633 +0000 UTC m=+2424.734045617" watchObservedRunningTime="2026-09-18 22:40:39.063477014 +0000 UTC m=+2424.734974984" Sep 18 22:40:45.053650 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:45.053616 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b86cc866f-r9nd9" Sep 18 22:40:45.525242 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:45.525199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:52.714928 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:52.714878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:40:53.091135 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:53.091026 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:40:53.091330 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:53.091283 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" podUID="410b93f6-9c02-4051-8e7a-6279102c0183" containerName="maas-api" containerID="cri-o://b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11" gracePeriod=30 Sep 18 22:40:54.340461 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.340433 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:40:54.395232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.395202 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8xnj\" (UniqueName: \"kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj\") pod \"410b93f6-9c02-4051-8e7a-6279102c0183\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " Sep 18 22:40:54.395232 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.395244 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca\") pod \"410b93f6-9c02-4051-8e7a-6279102c0183\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " Sep 18 22:40:54.395469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.395287 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs\") pod \"410b93f6-9c02-4051-8e7a-6279102c0183\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " Sep 18 22:40:54.395469 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.395319 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls\") pod \"410b93f6-9c02-4051-8e7a-6279102c0183\" (UID: \"410b93f6-9c02-4051-8e7a-6279102c0183\") " Sep 18 22:40:54.395699 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.395672 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "410b93f6-9c02-4051-8e7a-6279102c0183" (UID: "410b93f6-9c02-4051-8e7a-6279102c0183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 22:40:54.397753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.397720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "410b93f6-9c02-4051-8e7a-6279102c0183" (UID: "410b93f6-9c02-4051-8e7a-6279102c0183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:40:54.397753 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.397733 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj" (OuterVolumeSpecName: "kube-api-access-x8xnj") pod "410b93f6-9c02-4051-8e7a-6279102c0183" (UID: "410b93f6-9c02-4051-8e7a-6279102c0183"). InnerVolumeSpecName "kube-api-access-x8xnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 22:40:54.397983 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.397898 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "410b93f6-9c02-4051-8e7a-6279102c0183" (UID: "410b93f6-9c02-4051-8e7a-6279102c0183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 22:40:54.496436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.496403 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-metrics-certs\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:54.496436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.496432 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410b93f6-9c02-4051-8e7a-6279102c0183-maas-api-tls\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:54.496436 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.496441 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xnj\" (UniqueName: \"kubernetes.io/projected/410b93f6-9c02-4051-8e7a-6279102c0183-kube-api-access-x8xnj\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:54.496689 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:54.496450 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410b93f6-9c02-4051-8e7a-6279102c0183-openshift-service-ca\") on node \"ip-10-0-133-172.ec2.internal\" DevicePath \"\"" Sep 18 22:40:55.121641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.121537 2577 generic.go:356] "Generic (PLEG): container finished" podID="410b93f6-9c02-4051-8e7a-6279102c0183" containerID="b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11" exitCode=0 Sep 18 22:40:55.121641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.121578 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" Sep 18 22:40:55.121641 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.121621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" event={"ID":"410b93f6-9c02-4051-8e7a-6279102c0183","Type":"ContainerDied","Data":"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11"} Sep 18 22:40:55.121954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.121677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h" event={"ID":"410b93f6-9c02-4051-8e7a-6279102c0183","Type":"ContainerDied","Data":"b0b4a3bfa6a2084b7730822221703b69dbab6cf3f0b6215b0cb0c637149a699c"} Sep 18 22:40:55.121954 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.121699 2577 scope.go:117] "RemoveContainer" containerID="b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11" Sep 18 22:40:55.131125 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.131099 2577 scope.go:117] "RemoveContainer" containerID="b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11" Sep 18 22:40:55.131435 ip-10-0-133-172 kubenswrapper[2577]: E0918 22:40:55.131417 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11\": container with ID starting with b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11 not found: ID does not exist" containerID="b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11" Sep 18 22:40:55.131512 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.131445 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11"} err="failed to get container status \"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11\": rpc error: code = NotFound desc = could not find container \"b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11\": container with ID starting with b2070f2aa4cdabf259abdc593ab003fca49d662834e2f0a0a1162217083cea11 not found: ID does not exist" Sep 18 22:40:55.146481 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.146440 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:40:55.158608 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:55.158575 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-b89d7f5c7-tz79h"] Sep 18 22:40:56.848441 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:56.848403 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410b93f6-9c02-4051-8e7a-6279102c0183" path="/var/lib/kubelet/pods/410b93f6-9c02-4051-8e7a-6279102c0183/volumes" Sep 18 22:40:58.303294 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:40:58.303255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:41:05.303498 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:05.303455 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4kg8t"] Sep 18 22:41:06.472050 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:06.471999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:41:13.570914 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:13.570884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch_49479021-14f9-46e4-8185-9fe81e7fe5e0/copy-manifests/0.log" Sep 18 22:41:13.577333 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:13.577310 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch_49479021-14f9-46e4-8185-9fe81e7fe5e0/manager/0.log" Sep 18 22:41:14.275674 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:14.275646 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b86cc866f-r9nd9_88fb8fef-9ee8-472a-b844-37eb52ccb579/maas-api/0.log" Sep 18 22:41:15.936138 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:15.936110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4kg8t_b14ab5ad-fd10-4ce2-870d-0cc26b06baf5/limitador/0.log" Sep 18 22:41:16.743165 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:16.743131 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-666584dbdf-v2qxh_c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9/kube-auth-proxy/0.log" Sep 18 22:41:17.515310 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:17.515272 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5ml28_82829f42-260e-4cc1-badb-7df5b6e4b28a/main/0.log" Sep 18 22:41:17.739000 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:17.738969 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-dj2z6_b3092fc1-9de2-4c44-bca8-45173c8fa184/main/0.log" Sep 18 22:41:17.953601 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:17.953567 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbggvb_ede078d3-2454-4e28-baf7-fb9a25829f3a/main/0.log" Sep 18 22:41:18.172732 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:18.172698 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vnbrx_d7c0d48f-bba4-4ff1-965c-7340d07c8102/main/0.log" Sep 18 22:41:24.091846 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:24.091812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:41:24.190504 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:24.190461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" limit=1024 Sep 18 22:41:26.935682 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:26.935645 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dmkfl_d0405393-1b0b-446e-b958-231c5d2b0799/global-pull-secret-syncer/0.log" Sep 18 22:41:27.072754 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:27.072720 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2n8hq_59c036de-03e7-45e5-bf69-0f0c829f65b4/konnectivity-agent/0.log" Sep 18 22:41:27.162017 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:27.161957 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-172.ec2.internal_4ca949cbcee023acb5828904ff20c38f/haproxy/0.log" Sep 18 22:41:32.186207 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:32.186179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4kg8t_b14ab5ad-fd10-4ce2-870d-0cc26b06baf5/limitador/0.log" Sep 18 22:41:33.693519 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:33.693485 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-lw8ph_96343875-434b-4192-8130-794208b79619/cluster-monitoring-operator/0.log" Sep 18 22:41:33.803858 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:33.803767 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-774dc96fdf-k8sdr_2ddca12e-a916-47d6-adc9-358adf5d4c3c/metrics-server/0.log" Sep 18 22:41:33.873834 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:33.873774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfk2s_895fe44e-0d74-496e-98ce-0e4579ea112f/node-exporter/0.log" Sep 18 22:41:33.897740 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:33.897713 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfk2s_895fe44e-0d74-496e-98ce-0e4579ea112f/kube-rbac-proxy/0.log" Sep 18 22:41:33.921548 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:33.921531 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfk2s_895fe44e-0d74-496e-98ce-0e4579ea112f/init-textfile/0.log" Sep 18 22:41:34.274700 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.274671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/prometheus/0.log" Sep 18 22:41:34.298026 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.297999 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/config-reloader/0.log" Sep 18 22:41:34.321777 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.321730 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/thanos-sidecar/0.log" Sep 18 22:41:34.349303 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.349276 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/kube-rbac-proxy-web/0.log" Sep 18 22:41:34.374301 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.374221 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/kube-rbac-proxy/0.log" Sep 18 22:41:34.397505 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.397479 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/kube-rbac-proxy-thanos/0.log" Sep 18 22:41:34.421131 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.421094 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_694053ef-b656-42ca-a5b5-acc4f65078cb/init-config-reloader/0.log" Sep 18 22:41:34.453093 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.453058 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-wwpsf_66c566bf-133c-414f-ac85-81c8dd493eb8/prometheus-operator/0.log" Sep 18 22:41:34.477294 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.477257 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-wwpsf_66c566bf-133c-414f-ac85-81c8dd493eb8/kube-rbac-proxy/0.log" Sep 18 22:41:34.504374 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.504347 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9fqgf_5fbb370a-8423-401d-ad24-2b032c1db630/prometheus-operator-admission-webhook/0.log" Sep 18 22:41:34.638833 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.638725 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/thanos-query/0.log" Sep 18 22:41:34.662440 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.662403 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/kube-rbac-proxy-web/0.log" Sep 18 22:41:34.691492 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.691460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/kube-rbac-proxy/0.log" Sep 18 22:41:34.721477 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.721448 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/prom-label-proxy/0.log" Sep 18 22:41:34.749259 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.749233 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/kube-rbac-proxy-rules/0.log" Sep 18 22:41:34.801885 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:34.801849 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b86897f47-j56jw_a73b36d8-4f5c-4159-b744-220418518108/kube-rbac-proxy-metrics/0.log" Sep 18 22:41:36.353120 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:36.353094 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7jblm_e99cf92a-8040-4f12-8ba4-4b8801e1da30/networking-console-plugin/0.log" Sep 18 22:41:37.085910 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.085873 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54"] Sep 18 22:41:37.086381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.086351 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410b93f6-9c02-4051-8e7a-6279102c0183" containerName="maas-api" Sep 18 22:41:37.086381 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.086381 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="410b93f6-9c02-4051-8e7a-6279102c0183" containerName="maas-api" Sep 18 22:41:37.086588 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.086474 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="410b93f6-9c02-4051-8e7a-6279102c0183" containerName="maas-api" Sep 18 22:41:37.089900 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.089876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.092183 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.092160 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s5gh8\"/\"kube-root-ca.crt\"" Sep 18 22:41:37.093014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.092993 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s5gh8\"/\"openshift-service-ca.crt\"" Sep 18 22:41:37.093014 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.093011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s5gh8\"/\"default-dockercfg-mb95d\"" Sep 18 22:41:37.098033 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.098009 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54"] Sep 18 22:41:37.101633 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.101597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-sys\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.101810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.101659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-lib-modules\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.101810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.101698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-podres\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.101810 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.101753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-proc\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.101930 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.101896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk4n\" (UniqueName: \"kubernetes.io/projected/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-kube-api-access-6rk4n\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.202776 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-sys\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.202973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-lib-modules\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.202973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-podres\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.202973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-sys\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.202973 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-proc\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.203166 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.202989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-podres\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.203166 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.203004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-lib-modules\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.203166 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.203006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk4n\" (UniqueName: \"kubernetes.io/projected/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-kube-api-access-6rk4n\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.203166 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.203037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-proc\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.211711 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.211679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk4n\" (UniqueName: \"kubernetes.io/projected/d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3-kube-api-access-6rk4n\") pod \"perf-node-gather-daemonset-2fc54\" (UID: \"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3\") " pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.401410 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.401292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:37.677071 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.676922 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54"] Sep 18 22:41:37.680112 ip-10-0-133-172 kubenswrapper[2577]: W0918 22:41:37.680079 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd154cf2e_1ea7_4f8d_ba51_5a23c84f18b3.slice/crio-d1172c7571107188b3ee92c3cdf59f1d75a3cfe3c3a5035e41fdc504b2ac9ff2 WatchSource:0}: Error finding container d1172c7571107188b3ee92c3cdf59f1d75a3cfe3c3a5035e41fdc504b2ac9ff2: Status 404 returned error can't find the container with id d1172c7571107188b3ee92c3cdf59f1d75a3cfe3c3a5035e41fdc504b2ac9ff2 Sep 18 22:41:37.695745 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:37.695719 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-prtpp_df3b83c5-db34-4419-bfd6-3349890b09fc/download-server/0.log" Sep 18 22:41:38.340079 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:38.340042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" event={"ID":"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3","Type":"ContainerStarted","Data":"5a5a1716f850ef6cc72c911f3ad98f0b29ab13619ebfac3649b9db3c051b0a3e"} Sep 18 22:41:38.340282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:38.340094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" event={"ID":"d154cf2e-1ea7-4f8d-ba51-5a23c84f18b3","Type":"ContainerStarted","Data":"d1172c7571107188b3ee92c3cdf59f1d75a3cfe3c3a5035e41fdc504b2ac9ff2"} Sep 18 22:41:38.340282 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:38.340164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:38.360142 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:38.360055 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" podStartSLOduration=1.3600365810000001 podStartE2EDuration="1.360036581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:41:38.359363076 +0000 UTC m=+2484.030861044" watchObservedRunningTime="2026-09-18 22:41:38.360036581 +0000 UTC m=+2484.031534550" Sep 18 22:41:39.346992 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:39.346959 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdkwr_34307b55-2153-4ea8-914a-9c5fbffe1deb/dns/0.log" Sep 18 22:41:39.367681 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:39.367650 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdkwr_34307b55-2153-4ea8-914a-9c5fbffe1deb/kube-rbac-proxy/0.log" Sep 18 22:41:39.460883 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:39.460855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6wxg_33ac72fd-379b-4bc4-9fec-7942cf4ee26b/dns-node-resolver/0.log" Sep 18 22:41:40.142168 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:40.142134 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qltq_c03b8ee4-9b2b-4e88-8d60-fa6d6e597f92/node-ca/0.log" Sep 18 22:41:41.575665 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:41.575635 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-666584dbdf-v2qxh_c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9/kube-auth-proxy/0.log" Sep 18 22:41:42.687451 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:42.687421 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d6zt4_6e137694-e4b9-4aed-8de3-32e7f4984392/serve-healthcheck-canary/0.log" Sep 18 22:41:43.196605 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:43.196579 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2xnq2_092615d4-616e-43a9-87ef-747844e14570/kube-rbac-proxy/0.log" Sep 18 22:41:43.217870 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:43.217843 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2xnq2_092615d4-616e-43a9-87ef-747844e14570/exporter/0.log" Sep 18 22:41:43.239070 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:43.239041 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2xnq2_092615d4-616e-43a9-87ef-747844e14570/extractor/0.log" Sep 18 22:41:44.356089 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:44.356062 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s5gh8/perf-node-gather-daemonset-2fc54" Sep 18 22:41:46.038549 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:46.038439 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch_49479021-14f9-46e4-8185-9fe81e7fe5e0/manager/0.log" Sep 18 22:41:46.060582 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:46.060552 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79cf4c6fc4-fd7ch_49479021-14f9-46e4-8185-9fe81e7fe5e0/copy-manifests/0.log" Sep 18 22:41:47.609721 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:47.609689 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5fd7755fbb-nfqcl_fda16389-960b-4497-93da-6a882590cbee/manager/0.log" Sep 18 22:41:48.230391 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:48.230364 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-666584dbdf-v2qxh_c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9/kube-auth-proxy/0.log" Sep 18 22:41:50.188771 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:50.188739 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-666584dbdf-v2qxh_c5cb7e18-5fa7-4ed6-aa4e-70fe8b9025a9/kube-auth-proxy/0.log" Sep 18 22:41:54.363980 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.363943 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/kube-multus-additional-cni-plugins/0.log" Sep 18 22:41:54.387547 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.387516 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/egress-router-binary-copy/0.log" Sep 18 22:41:54.409709 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.409670 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/cni-plugins/0.log" Sep 18 22:41:54.431800 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.431754 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/bond-cni-plugin/0.log" Sep 18 22:41:54.454704 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.454669 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/routeoverride-cni/0.log" Sep 18 22:41:54.476713 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.476686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/whereabouts-cni-bincopy/0.log" Sep 18 22:41:54.496971 ip-10-0-133-172 kubenswrapper[2577]: I0918 22:41:54.496944 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l29z8_96d1fbbd-29b7-4e76-b52e-b504c0ca9602/whereabouts-cni/0.log"