Jul 18 11:49:06.102927 ip-10-0-137-172 systemd[1]: Starting Kubernetes Kubelet... Jul 18 11:49:06.429706 ip-10-0-137-172 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:49:06.429706 ip-10-0-137-172 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 11:49:06.429706 ip-10-0-137-172 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:49:06.429706 ip-10-0-137-172 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 11:49:06.429706 ip-10-0-137-172 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:49:06.433273 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.433158 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442199 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442219 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442223 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442227 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442231 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.442228 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442234 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442237 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442241 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442244 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442248 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442250 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442253 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442256 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442259 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442262 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442265 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442267 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442270 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442272 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442275 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442277 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442280 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442282 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442285 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442291 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.442453 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442294 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442296 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442299 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442302 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442304 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442307 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442309 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442311 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442314 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442317 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442319 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442321 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442324 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442326 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442329 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442332 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442335 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442337 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442340 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442343 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.442947 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442345 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442348 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442350 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442352 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442355 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442358 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442360 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442363 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442365 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442370 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442374 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442378 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442383 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442386 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442389 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442391 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442394 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442397 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442399 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.443433 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442402 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442405 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442407 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442409 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442412 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442414 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442417 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442420 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442423 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442425 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442428 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442430 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442433 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442435 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442438 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442440 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442443 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442445 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442448 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442452 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.443913 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442454 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442457 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442459 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442464 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442466 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442469 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442471 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442474 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442476 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442479 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442481 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442485 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442489 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442492 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.442495 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443246 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443254 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443258 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443261 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.444409 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443264 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443266 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443269 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443272 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443274 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443277 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443280 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443282 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443285 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443288 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443290 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443293 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443295 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443299 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443303 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443306 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443309 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443311 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443314 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443316 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.444881 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443319 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443322 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443324 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443327 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443329 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443331 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443336 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443340 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443343 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443346 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443353 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443355 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443358 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443360 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443363 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443365 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443368 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443370 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443373 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.445386 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443375 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443378 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443381 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443383 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443387 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443389 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443392 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443395 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443397 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443400 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443402 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443404 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443407 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443409 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443412 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443414 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443416 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443419 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443421 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443424 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.445889 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443426 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443429 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443431 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443435 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443437 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443440 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443443 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443445 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443448 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443450 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443453 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443455 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443458 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443460 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443463 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443465 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443467 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443470 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443472 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443474 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.446371 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443477 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443479 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443482 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443484 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443487 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443490 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443492 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443494 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443498 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443501 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443504 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443508 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443511 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443514 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443517 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443520 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443930 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443936 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443939 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.446870 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443942 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443944 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443947 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443950 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443952 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443955 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443960 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443964 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443967 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443969 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443972 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443975 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443978 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443980 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443983 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443985 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443988 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443990 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443993 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.447325 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.443996 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444000 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444002 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444005 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444007 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444009 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444012 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444015 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444018 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444022 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444024 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444027 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444030 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444032 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444035 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444037 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444040 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444042 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444045 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444047 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.447837 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444050 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444053 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444055 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444057 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444060 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444062 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444065 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444067 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444070 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444072 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444074 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444077 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444079 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444082 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444084 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444086 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444089 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444091 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444093 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444096 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.448321 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444098 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444101 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444104 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444106 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444109 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444112 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444114 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444116 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444119 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444121 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444124 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444126 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444129 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444131 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444134 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444136 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444139 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444143 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444147 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.448831 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444149 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444153 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444156 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444159 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444162 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444164 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444167 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444170 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444172 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444175 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444178 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444180 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444183 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444186 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444189 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444192 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444195 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444197 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444270 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444277 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444283 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 11:49:06.449339 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444288 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444292 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444295 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444299 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444304 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444307 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444309 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444313 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444316 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444319 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444322 2573 flags.go:64] FLAG: --cgroup-root="" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444324 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444327 2573 flags.go:64] FLAG: --client-ca-file="" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444330 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444333 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444337 2573 flags.go:64] FLAG: --cluster-domain="" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444339 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444342 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444345 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444348 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444352 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444355 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444358 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 11:49:06.449864 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444361 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444364 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444368 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444371 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444373 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444376 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444380 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444383 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444385 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444388 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444391 2573 flags.go:64] FLAG: --enable-server="true" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444394 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444398 2573 flags.go:64] FLAG: --event-burst="100" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444401 2573 flags.go:64] FLAG: --event-qps="50" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444404 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444407 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444410 2573 flags.go:64] FLAG: --eviction-hard="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444414 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444416 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444419 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444422 2573 flags.go:64] FLAG: --eviction-soft="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444425 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444428 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444431 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444434 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 11:49:06.450420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444436 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444439 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444442 2573 flags.go:64] FLAG: --feature-gates="" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444445 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444449 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444451 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444454 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444457 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444460 2573 flags.go:64] FLAG: --help="false" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444463 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444466 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444469 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444471 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444474 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444477 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444480 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444483 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444486 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444490 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444492 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444495 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444498 2573 flags.go:64] FLAG: --kube-reserved="" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444501 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444503 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 11:49:06.451118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444506 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444509 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444511 2573 flags.go:64] FLAG: --lock-file="" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444514 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444517 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444519 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444525 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444528 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444530 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444533 2573 flags.go:64] FLAG: --logging-format="text" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444536 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444539 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444541 2573 flags.go:64] FLAG: --manifest-url="" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444544 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444548 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444550 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444554 2573 flags.go:64] FLAG: --max-pods="110" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444557 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444560 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444563 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444565 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444568 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444571 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444574 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444581 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 11:49:06.451727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444584 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444588 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444592 2573 flags.go:64] FLAG: --pod-cidr="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444595 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444600 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444603 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444605 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444608 2573 flags.go:64] FLAG: --port="10250" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444611 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444614 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd41e74d9cd73eb4" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444617 2573 flags.go:64] FLAG: --qos-reserved="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444620 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444623 2573 flags.go:64] FLAG: --register-node="true" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444625 2573 flags.go:64] FLAG: --register-with-taints="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444629 2573 flags.go:64] FLAG: --registry-burst="10" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444632 2573 flags.go:64] FLAG: --registry-qps="5" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444635 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444638 2573 flags.go:64] FLAG: --reserved-memory="" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444641 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444644 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444647 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444649 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444652 2573 flags.go:64] FLAG: --runonce="false" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444655 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444658 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 11:49:06.452322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444660 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444663 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444666 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444683 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444686 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444689 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444692 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444694 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444699 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444701 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444704 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444707 2573 flags.go:64] FLAG: --system-cgroups="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444709 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444714 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444717 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444720 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444723 2573 flags.go:64] FLAG: --tls-min-version="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444726 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444728 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444731 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444734 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444736 2573 flags.go:64] FLAG: --v="2" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444740 2573 flags.go:64] FLAG: --version="false" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444745 2573 flags.go:64] FLAG: --vmodule="" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444754 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 11:49:06.452933 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.444758 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444864 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444868 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444871 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444875 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444878 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444881 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444884 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444887 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444891 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444894 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444897 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444900 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444903 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444905 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444910 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444912 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444915 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444917 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.453527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444919 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444922 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444924 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444927 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444929 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444931 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444934 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444936 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444939 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444941 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444944 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444946 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444949 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444951 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444954 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444956 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444959 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444961 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444963 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444966 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.454108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444969 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444972 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444974 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444977 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444979 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444982 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444984 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444987 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444990 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444993 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444995 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.444998 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445000 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445002 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445005 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445007 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445009 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445012 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445014 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.454601 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445016 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445019 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445022 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445026 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445029 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445031 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445037 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445039 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445042 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445044 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445047 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445049 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445052 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445055 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445058 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445060 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445063 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445065 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445067 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445070 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.455077 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445072 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445076 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445078 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445081 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445083 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445086 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445088 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445090 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445093 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445095 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445098 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445100 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445103 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445105 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445108 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445111 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445113 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445115 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445118 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445120 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.455599 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445123 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.445126 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.445603 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452198 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452217 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452236 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452242 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452302 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452306 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452310 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452313 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452316 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452319 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452322 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452325 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.456105 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452328 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452331 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452334 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452337 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452341 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452345 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452348 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452351 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452354 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452357 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452359 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452362 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452365 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452368 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452371 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452373 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452376 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452380 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.456558 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452384 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452387 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452390 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452394 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452396 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452399 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452402 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452405 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452407 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452410 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452413 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452416 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452418 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452421 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452424 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452441 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452445 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452448 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452450 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452453 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.457050 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452456 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452459 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452462 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452465 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452467 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452470 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452473 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452475 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452478 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452480 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452483 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452486 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452488 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452493 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452496 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452499 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452502 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452505 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452507 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.457572 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452510 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452512 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452515 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452517 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452520 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452522 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452524 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452527 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452529 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452531 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452534 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452536 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452538 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452541 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452543 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452546 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452548 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452551 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452553 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452555 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.458089 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452558 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452560 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452563 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452565 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452568 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452571 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452573 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452576 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452579 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452581 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452585 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452587 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452590 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452592 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452597 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452732 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:49:06.458778 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452739 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452742 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452744 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452747 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452750 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452753 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452755 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452757 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452760 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452762 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452765 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452767 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452770 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452772 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452775 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452777 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452781 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452785 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452788 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452790 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:49:06.459211 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452793 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452796 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452799 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452802 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452805 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452808 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452810 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452813 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452816 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452818 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452820 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452823 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452826 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452828 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452831 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452833 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452835 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452838 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452840 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452843 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:49:06.459879 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452846 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452848 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452851 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452853 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452856 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452858 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452860 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452863 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452865 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452867 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452870 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452872 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452874 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452877 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452880 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452883 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452885 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452888 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452890 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452892 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:49:06.460527 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452896 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452898 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452901 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452903 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452906 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452909 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452912 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452914 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452917 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452919 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452922 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452924 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452926 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452929 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452931 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452934 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452936 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452939 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452941 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:49:06.461223 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452944 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452948 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452952 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452955 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452957 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452960 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452963 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452965 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452968 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452971 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452973 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452976 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452978 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452980 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452983 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452985 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452988 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452990 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:49:06.461886 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:06.452992 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.452997 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.453099 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.454924 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.456019 2573 server.go:1019] "Starting client certificate rotation" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.456135 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.456173 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.458159 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 11:49:06.462318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.460762 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 11:49:06.470344 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.470323 2573 log.go:25] "Validated CRI v1 runtime API" Jul 18 11:49:06.473988 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.473972 2573 log.go:25] "Validated CRI v1 image API" Jul 18 11:49:06.474789 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.474775 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 11:49:06.476750 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.476732 2573 fs.go:135] Filesystem UUIDs: map[151f4ad9-bfb4-446b-93d7-78230c179116:/dev/nvme0n1p4 5e6bb27c-e9c8-49b6-a8a0-987c5a78d03e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 18 11:49:06.476818 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.476750 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 11:49:06.481526 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.481422 2573 manager.go:216] Machine: {Timestamp:2026-07-18 11:49:06.48027861 +0000 UTC m=+0.281087461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100828 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29caa7741f4038d690b778e8d76031 SystemUUID:ec29caa7-741f-4038-d690-b778e8d76031 BootID:20337e4a-180a-4d1c-a4be-f8eb4e5843ad Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:94:fb:63:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:94:fb:63:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:96:11:7d:dd:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 11:49:06.481526 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.481520 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 11:49:06.481647 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.481602 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 11:49:06.484341 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.484311 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 11:49:06.484477 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.484343 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 11:49:06.484958 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.484948 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 11:49:06.484991 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.484962 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 11:49:06.484991 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.484969 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 11:49:06.485615 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.485604 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 11:49:06.485943 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.485917 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 11:49:06.486963 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.486950 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 11:49:06.490044 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490031 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 18 11:49:06.490177 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490165 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 11:49:06.490236 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490204 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 18 11:49:06.490236 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490218 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 11:49:06.490318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490237 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 11:49:06.490318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490252 2573 kubelet.go:398] "Adding apiserver pod source" Jul 18 11:49:06.490318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490266 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 11:49:06.490623 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490609 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 11:49:06.490715 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.490632 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 11:49:06.492510 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.492491 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 11:49:06.494128 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.494115 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 11:49:06.494176 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.494145 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 11:49:06.495036 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495024 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495040 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495046 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495051 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495060 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495065 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495073 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495088 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495094 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 11:49:06.495103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495099 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 11:49:06.495332 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495112 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 11:49:06.495332 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.495118 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 11:49:06.496283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.496273 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 11:49:06.496283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.496283 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 11:49:06.499266 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.499238 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 18 11:49:06.499344 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.499283 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 11:49:06.499344 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.499302 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 11:49:06.499427 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.499414 2573 server.go:1268] "Started kubelet" Jul 18 11:49:06.499524 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.499479 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 11:49:06.499571 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.499489 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 11:49:06.499613 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.499576 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 11:49:06.502333 ip-10-0-137-172 systemd[1]: Started Kubernetes Kubelet. Jul 18 11:49:06.504784 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.503939 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 18 11:49:06.505763 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.505741 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 11:49:06.508628 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.507825 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18c35fec720f89d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-07-18 11:49:06.499389913 +0000 UTC m=+0.300198764,LastTimestamp:2026-07-18 11:49:06.499389913 +0000 UTC m=+0.300198764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Jul 18 11:49:06.508798 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.508778 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 11:49:06.510755 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.510735 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 11:49:06.510863 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.510738 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 11:49:06.510863 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.510805 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.511389 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511371 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 11:49:06.511468 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511393 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 11:49:06.511468 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511371 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 11:49:06.511549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511515 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 18 11:49:06.511549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511525 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 18 11:49:06.511641 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511622 2573 factory.go:55] Registering systemd factory Jul 18 11:49:06.511710 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511646 2573 factory.go:223] Registration of the systemd container factory successfully Jul 18 11:49:06.511882 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511870 2573 factory.go:153] Registering CRI-O factory Jul 18 11:49:06.511949 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511884 2573 factory.go:223] Registration of the crio container factory successfully Jul 18 11:49:06.511949 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511934 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 11:49:06.512019 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511959 2573 factory.go:103] Registering Raw factory Jul 18 11:49:06.512019 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.511979 2573 manager.go:1196] Started watching for new ooms in manager Jul 18 11:49:06.512385 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.512374 2573 manager.go:349] Starting recovery of all containers Jul 18 11:49:06.514705 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.514659 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4mf92" Jul 18 11:49:06.516318 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.516290 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 18 11:49:06.516318 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.516301 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 18 11:49:06.519365 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.519336 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 11:49:06.521746 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.521727 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4mf92" Jul 18 11:49:06.522908 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.522892 2573 manager.go:354] Recovery completed Jul 18 11:49:06.527542 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.527529 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.529325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529307 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.529402 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529341 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.529402 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529355 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.529903 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529889 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 11:49:06.529903 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529901 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 11:49:06.530025 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529915 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 18 11:49:06.530025 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.529930 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.530628 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.530614 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.530747 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.530647 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.530799 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.530781 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.531024 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.531011 2573 policy_none.go:49] "None policy: Start" Jul 18 11:49:06.531024 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.531026 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 11:49:06.531124 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.531036 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 11:49:06.531124 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.531048 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18c35fec73d851d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-07-18 11:49:06.529325525 +0000 UTC m=+0.330134382,LastTimestamp:2026-07-18 11:49:06.529325525 +0000 UTC m=+0.330134382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Jul 18 11:49:06.532020 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.532005 2573 policy_none.go:47] "Start" Jul 18 11:49:06.575359 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575218 2573 manager.go:337] "Starting Device Plugin manager" Jul 18 11:49:06.575471 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.575409 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 11:49:06.575471 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575423 2573 server.go:87] "Starting device plugin registration server" Jul 18 11:49:06.575739 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575727 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 11:49:06.575779 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575743 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 11:49:06.575865 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575829 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 11:49:06.575954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575932 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 11:49:06.575954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.575942 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 11:49:06.576427 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.576406 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 11:49:06.576503 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.576458 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.638719 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.638693 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 11:49:06.639738 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.638733 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 11:49:06.639738 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.638754 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 11:49:06.639738 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.638784 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 11:49:06.640917 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.640899 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:49:06.676240 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.676217 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.677634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.677614 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.677743 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.677650 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.677743 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.677667 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.677743 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.677714 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.685761 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.685708 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.685761 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.685734 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-172.ec2.internal\": node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.699485 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.699461 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.739370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.739341 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"] Jul 18 11:49:06.739497 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.739420 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.740383 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.740367 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.740452 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.740394 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.740452 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.740404 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.741559 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.741540 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.741666 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.741651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.741729 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.741695 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.742260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742237 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.742338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742268 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.742338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742278 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.742338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742237 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.742428 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742348 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.742428 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.742359 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.743287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.743273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.743334 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.743299 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:49:06.744070 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.744049 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:49:06.744152 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.744075 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:49:06.744152 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.744087 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:49:06.767211 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.767191 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.771493 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.771477 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.799974 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.799947 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.812958 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.812929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.812958 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.812959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.813111 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.812975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.900940 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:06.900912 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:06.913298 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.913298 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.913427 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.913427 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.913427 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913377 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:06.913427 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:06.913417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:07.001804 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.001717 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.070211 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.070179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:07.074010 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.073865 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:07.102347 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.102313 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.202839 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.202800 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.303356 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.303289 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.403952 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.403923 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.455509 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.455478 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 11:49:07.455954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.455613 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 11:49:07.504370 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:07.504346 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Jul 18 11:49:07.525902 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.525863 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 11:44:07 +0000 UTC" deadline="2028-01-29 07:15:57.466108998 +0000 UTC" Jul 18 11:49:07.525902 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:07.525894 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13435h26m49.940218417s" Jul 18 11:49:08.895368 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.895284 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:49:08.906112 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:08.906084 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deef81c0d8c2f4458539863e90bc2e7.slice/crio-fd6bdc55f9890b2956d37f7e4fa4f7c585a0fc53d566e2c0f0e5a811bfbf5d3d WatchSource:0}: Error finding container fd6bdc55f9890b2956d37f7e4fa4f7c585a0fc53d566e2c0f0e5a811bfbf5d3d: Status 404 returned error can't find the container with id fd6bdc55f9890b2956d37f7e4fa4f7c585a0fc53d566e2c0f0e5a811bfbf5d3d Jul 18 11:49:08.906808 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:08.906793 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb9b55189c2ca1c136d2bbd25cb800a.slice/crio-c931c71b42ec59ea10586f1a4426fd88be94f2846355238cb598b401663325d1 WatchSource:0}: Error finding container c931c71b42ec59ea10586f1a4426fd88be94f2846355238cb598b401663325d1: Status 404 returned error can't find the container with id c931c71b42ec59ea10586f1a4426fd88be94f2846355238cb598b401663325d1 Jul 18 11:49:08.910638 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.910624 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:49:08.921534 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.921514 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Jul 18 11:49:08.931784 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.931769 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 11:49:08.933871 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.933859 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Jul 18 11:49:08.940245 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.940222 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 11:49:08.952625 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.952578 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"bbb9b55189c2ca1c136d2bbd25cb800a","Type":"ContainerStarted","Data":"c931c71b42ec59ea10586f1a4426fd88be94f2846355238cb598b401663325d1"} Jul 18 11:49:08.953591 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:08.953559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerStarted","Data":"fd6bdc55f9890b2956d37f7e4fa4f7c585a0fc53d566e2c0f0e5a811bfbf5d3d"} Jul 18 11:49:09.266544 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.266462 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:49:09.291775 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.291754 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:49:09.682777 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.682684 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:49:09.801364 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.801329 2573 apiserver.go:52] "Watching apiserver" Jul 18 11:49:09.808098 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.808056 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 11:49:09.809576 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.809545 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vfgt4","kube-system/global-pull-secret-syncer-xfghz","kube-system/konnectivity-agent-59zmj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2","openshift-multus/multus-5rqww","openshift-multus/multus-additional-cni-plugins-xcss9","openshift-multus/network-metrics-daemon-qrb7g","openshift-network-operator/iptables-alerter-7gmlr","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qrlm8","openshift-dns/node-resolver-t7ntd","openshift-image-registry/node-ca-j7d7p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","openshift-network-diagnostics/network-check-target-6gtc8"] Jul 18 11:49:09.810985 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.810967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.812471 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.812453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.813096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.813075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 11:49:09.813186 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.813083 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g7rnt\"" Jul 18 11:49:09.813385 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.813368 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.813469 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.813381 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.813761 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.813746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.813859 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.813820 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:09.815514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.814827 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-26bjq\"" Jul 18 11:49:09.815514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.814838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.815514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.814894 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 11:49:09.815514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.814966 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.815514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.815233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.816194 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.816173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.817176 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.817156 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-748rq\"" Jul 18 11:49:09.817299 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.817184 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 11:49:09.817299 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.817240 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 11:49:09.817506 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.817491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.818479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.818049 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.818479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.818434 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdktk\"" Jul 18 11:49:09.818479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.818473 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 11:49:09.818643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.818575 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 11:49:09.818800 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.818784 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.819400 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.819370 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 11:49:09.819488 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.819466 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 11:49:09.819488 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.819387 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l6wq9\"" Jul 18 11:49:09.820077 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.820056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.821383 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.821366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.821474 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.821437 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:09.822001 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.821985 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 11:49:09.822213 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.822197 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.822431 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.822418 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 11:49:09.822611 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.822596 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.822837 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.822821 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5pnxg\"" Jul 18 11:49:09.822997 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.822985 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 11:49:09.823107 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.823094 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 11:49:09.823346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.823329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.823440 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.823352 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 11:49:09.824345 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.824325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:09.825402 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.825384 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.825520 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.825503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:09.825878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.825762 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.825878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.825779 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g6gg9\"" Jul 18 11:49:09.826495 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.826080 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.826495 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.826167 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.826495 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.826339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m2rxv\"" Jul 18 11:49:09.827474 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.827456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:09.827554 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.827539 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 11:49:09.827612 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.827539 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:09.827692 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.827675 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l6jcv\"" Jul 18 11:49:09.827941 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.827925 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 11:49:09.828101 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.828085 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-modprobe-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-device-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad3d50c-69ad-4920-9320-2bd666d79a82-host-slash\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-os-release\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-multus-certs\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-cnibin\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmng\" (UniqueName: \"kubernetes.io/projected/15e43a5d-171f-4982-b708-de1f8ef41752-kube-api-access-vdmng\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfjq\" (UniqueName: \"kubernetes.io/projected/05c53186-396a-42a6-84c8-2d0c48c5237f-kube-api-access-vlfjq\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cnibin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-netns\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-var-lib-kubelet\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8tq\" (UniqueName: \"kubernetes.io/projected/1ad3d50c-69ad-4920-9320-2bd666d79a82-kube-api-access-vd8tq\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.832027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830443 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4v2m\" (UniqueName: \"kubernetes.io/projected/afd98e56-e223-4c37-92bb-7a80e4a92e14-kube-api-access-x4v2m\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-node-log\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysconfig\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-systemd\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-host\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-tmp\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-hostroot\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-system-cni-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-run\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-agent-certs\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-systemd-units\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-netns\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-socket-dir-parent\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbxv\" (UniqueName: \"kubernetes.io/projected/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-kube-api-access-kbbxv\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.832577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-script-lib\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-sys-fs\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad3d50c-69ad-4920-9320-2bd666d79a82-iptables-alerter-script\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-kubelet\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-dbus\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-sys\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-systemd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-log-socket\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-env-overrides\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-k8s-cni-cncf-io\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-var-lib-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4nc\" (UniqueName: \"kubernetes.io/projected/3b052873-a80a-443c-8a2e-88bca2b4f57f-kube-api-access-dn4nc\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.830992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-lib-modules\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528wj\" (UniqueName: \"kubernetes.io/projected/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-kube-api-access-528wj\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-socket-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.833287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-system-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-etc-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-kubernetes\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-tuned\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-registration-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-slash\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-ovn\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-conf\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-konnectivity-ca\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cni-binary-copy\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-multus\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-conf-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-daemon-config\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.833964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-os-release\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-kubelet\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-bin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-etc-kubernetes\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-config\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-bin\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-netd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e43a5d-171f-4982-b708-de1f8ef41752-ovn-node-metrics-cert\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-kubelet-config\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.834643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.831463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.842987 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.842968 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 11:49:09.853972 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.853946 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8wmd" Jul 18 11:49:09.861277 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.861251 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8wmd" Jul 18 11:49:09.923070 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.923044 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 11:49:09.932348 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-socket-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-system-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-tmp-dir\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-etc-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-kubernetes\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-tuned\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-system-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.932500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-registration-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-etc-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.932545 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-socket-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-registration-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932649 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.932663 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.432638533 +0000 UTC m=+2.923410649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-slash\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-ovn\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-kubernetes\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-conf\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-ovn\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932855 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-slash\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f205190-9537-4e79-a19b-3b796d3fda6b-serviceca\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:09.932936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-konnectivity-ca\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-conf\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cni-binary-copy\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.932996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-multus\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-conf-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-daemon-config\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-os-release\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-multus\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-kubelet\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-bin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-os-release\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-etc-kubernetes\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-kubelet\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-config\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.933742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-cni-bin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933536 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-bin\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-konnectivity-ca\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cni-binary-copy\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-netd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933587 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-etc-kubernetes\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e43a5d-171f-4982-b708-de1f8ef41752-ovn-node-metrics-cert\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-bin\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933624 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-cni-netd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933655 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-daemon-config\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-kubelet-config\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-conf-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-kubelet-config\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.934442 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-modprobe-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-device-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933854 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-device-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad3d50c-69ad-4920-9320-2bd666d79a82-host-slash\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-os-release\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-modprobe-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-multus-certs\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad3d50c-69ad-4920-9320-2bd666d79a82-host-slash\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.933996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-os-release\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-cnibin\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-multus-certs\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-cnibin\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmng\" (UniqueName: \"kubernetes.io/projected/15e43a5d-171f-4982-b708-de1f8ef41752-kube-api-access-vdmng\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-config\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.935287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfjq\" (UniqueName: \"kubernetes.io/projected/05c53186-396a-42a6-84c8-2d0c48c5237f-kube-api-access-vlfjq\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cnibin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.934184 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd98e56-e223-4c37-92bb-7a80e4a92e14-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-netns\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-var-lib-kubelet\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:09.934235 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.434223471 +0000 UTC m=+2.924995566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-netns\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8w7\" (UniqueName: \"kubernetes.io/projected/5f205190-9537-4e79-a19b-3b796d3fda6b-kube-api-access-hr8w7\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-var-lib-kubelet\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-hosts-file\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-cnibin\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mml8p\" (UniqueName: \"kubernetes.io/projected/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-kube-api-access-mml8p\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8tq\" (UniqueName: \"kubernetes.io/projected/1ad3d50c-69ad-4920-9320-2bd666d79a82-kube-api-access-vd8tq\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934437 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 11:49:09.936110 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4v2m\" (UniqueName: \"kubernetes.io/projected/afd98e56-e223-4c37-92bb-7a80e4a92e14-kube-api-access-x4v2m\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-node-log\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-node-log\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysconfig\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-systemd\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-host\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-systemd\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-tmp\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysconfig\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-host\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-hostroot\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-system-cni-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-run\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934844 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-cni-dir\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd98e56-e223-4c37-92bb-7a80e4a92e14-system-cni-dir\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.936762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-hostroot\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-agent-certs\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-systemd-units\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-netns\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934911 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-run\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.934986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f205190-9537-4e79-a19b-3b796d3fda6b-host\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-systemd-units\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-socket-dir-parent\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-host-run-netns\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbxv\" (UniqueName: \"kubernetes.io/projected/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-kube-api-access-kbbxv\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-script-lib\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-multus-socket-dir-parent\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-sys-fs\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad3d50c-69ad-4920-9320-2bd666d79a82-iptables-alerter-script\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-kubelet\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.937453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-dbus\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-sys\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-systemd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-log-socket\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-var-lib-kubelet\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-env-overrides\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-k8s-cni-cncf-io\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05c53186-396a-42a6-84c8-2d0c48c5237f-sys-fs\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-var-lib-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4nc\" (UniqueName: \"kubernetes.io/projected/3b052873-a80a-443c-8a2e-88bca2b4f57f-kube-api-access-dn4nc\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-lib-modules\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528wj\" (UniqueName: \"kubernetes.io/projected/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-kube-api-access-528wj\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-ovnkube-script-lib\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935550 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-var-lib-openvswitch\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15e43a5d-171f-4982-b708-de1f8ef41752-env-overrides\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935853 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-sysctl-d\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-host-run-k8s-cni-cncf-io\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-dbus\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-run-systemd\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-lib-modules\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-sys\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.935958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15e43a5d-171f-4982-b708-de1f8ef41752-log-socket\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.936062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad3d50c-69ad-4920-9320-2bd666d79a82-iptables-alerter-script\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.938209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-etc-tuned\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.938292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15e43a5d-171f-4982-b708-de1f8ef41752-ovn-node-metrics-cert\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.938749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.938482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b2d2ab6-93aa-40bb-817e-ef96789eb1b5-agent-certs\") pod \"konnectivity-agent-59zmj\" (UID: \"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5\") " pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:09.939138 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.938837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-tmp\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.944691 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.944667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8tq\" (UniqueName: \"kubernetes.io/projected/1ad3d50c-69ad-4920-9320-2bd666d79a82-kube-api-access-vd8tq\") pod \"iptables-alerter-7gmlr\" (UID: \"1ad3d50c-69ad-4920-9320-2bd666d79a82\") " pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:09.945143 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.945121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmng\" (UniqueName: \"kubernetes.io/projected/15e43a5d-171f-4982-b708-de1f8ef41752-kube-api-access-vdmng\") pod \"ovnkube-node-vfgt4\" (UID: \"15e43a5d-171f-4982-b708-de1f8ef41752\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:09.955721 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.952729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbxv\" (UniqueName: \"kubernetes.io/projected/1866a7a6-57c5-4393-a7bf-6d3cd53e21f1-kube-api-access-kbbxv\") pod \"multus-5rqww\" (UID: \"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1\") " pod="openshift-multus/multus-5rqww" Jul 18 11:49:09.955721 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.952729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4nc\" (UniqueName: \"kubernetes.io/projected/3b052873-a80a-443c-8a2e-88bca2b4f57f-kube-api-access-dn4nc\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:09.955721 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.954874 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4v2m\" (UniqueName: \"kubernetes.io/projected/afd98e56-e223-4c37-92bb-7a80e4a92e14-kube-api-access-x4v2m\") pod \"multus-additional-cni-plugins-xcss9\" (UID: \"afd98e56-e223-4c37-92bb-7a80e4a92e14\") " pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:09.956780 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.956755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528wj\" (UniqueName: \"kubernetes.io/projected/f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf-kube-api-access-528wj\") pod \"tuned-qrlm8\" (UID: \"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf\") " pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:09.957992 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:09.957965 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfjq\" (UniqueName: \"kubernetes.io/projected/05c53186-396a-42a6-84c8-2d0c48c5237f-kube-api-access-vlfjq\") pod \"aws-ebs-csi-driver-node-xs9m2\" (UID: \"05c53186-396a-42a6-84c8-2d0c48c5237f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:10.036155 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f205190-9537-4e79-a19b-3b796d3fda6b-serviceca\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8w7\" (UniqueName: \"kubernetes.io/projected/5f205190-9537-4e79-a19b-3b796d3fda6b-kube-api-access-hr8w7\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-hosts-file\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mml8p\" (UniqueName: \"kubernetes.io/projected/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-kube-api-access-mml8p\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f205190-9537-4e79-a19b-3b796d3fda6b-host\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.036325 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036304 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-hosts-file\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.036585 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-tmp-dir\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.036585 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f205190-9537-4e79-a19b-3b796d3fda6b-host\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.036585 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-tmp-dir\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.036703 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.036685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f205190-9537-4e79-a19b-3b796d3fda6b-serviceca\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.042895 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.042863 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:10.042895 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.042887 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:10.042895 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.042901 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.043101 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.042957 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.542943959 +0000 UTC m=+3.033716055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.044723 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.044689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mml8p\" (UniqueName: \"kubernetes.io/projected/d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f-kube-api-access-mml8p\") pod \"node-resolver-t7ntd\" (UID: \"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f\") " pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.045336 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.045314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8w7\" (UniqueName: \"kubernetes.io/projected/5f205190-9537-4e79-a19b-3b796d3fda6b-kube-api-access-hr8w7\") pod \"node-ca-j7d7p\" (UID: \"5f205190-9537-4e79-a19b-3b796d3fda6b\") " pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.122624 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.122591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" Jul 18 11:49:10.132679 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.132654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gmlr" Jul 18 11:49:10.144409 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.144386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:10.148996 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.148972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rqww" Jul 18 11:49:10.155727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.155693 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcss9" Jul 18 11:49:10.165675 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.165653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:10.173482 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.173463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" Jul 18 11:49:10.180788 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.180761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t7ntd" Jul 18 11:49:10.188195 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.188169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j7d7p" Jul 18 11:49:10.439656 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.439577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:10.439656 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.439629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:10.439890 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.439755 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:10.439890 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.439767 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:10.439890 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.439812 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:11.439798615 +0000 UTC m=+3.930570725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:10.439890 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.439827 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:11.439821124 +0000 UTC m=+3.930593219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:10.641114 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.641069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:10.641275 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.641251 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:10.641314 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.641279 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:10.641314 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.641291 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.641394 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:10.641342 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:11.641328479 +0000 UTC m=+4.132100576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.788974 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:10.788945 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c53186_396a_42a6_84c8_2d0c48c5237f.slice/crio-908ceb02658b1ecd8dab37ff09b9c395a76e72dcf07ebd8dc6a9b4597260f42d WatchSource:0}: Error finding container 908ceb02658b1ecd8dab37ff09b9c395a76e72dcf07ebd8dc6a9b4597260f42d: Status 404 returned error can't find the container with id 908ceb02658b1ecd8dab37ff09b9c395a76e72dcf07ebd8dc6a9b4597260f42d Jul 18 11:49:10.792063 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:10.792038 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2d2ab6_93aa_40bb_817e_ef96789eb1b5.slice/crio-b988aed991b9bb07b353170f47774d6272946c00eb509ea9c21bf99f5f725e22 WatchSource:0}: Error finding container b988aed991b9bb07b353170f47774d6272946c00eb509ea9c21bf99f5f725e22: Status 404 returned error can't find the container with id b988aed991b9bb07b353170f47774d6272946c00eb509ea9c21bf99f5f725e22 Jul 18 11:49:10.793624 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:10.793601 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd98e56_e223_4c37_92bb_7a80e4a92e14.slice/crio-1b5f8dd8cbd8312b5a86147306764cdc0dc30f67c059bff2a5381db887d1d962 WatchSource:0}: Error finding container 1b5f8dd8cbd8312b5a86147306764cdc0dc30f67c059bff2a5381db887d1d962: Status 404 returned error can't find the container with id 1b5f8dd8cbd8312b5a86147306764cdc0dc30f67c059bff2a5381db887d1d962 Jul 18 11:49:10.794595 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:10.794549 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad3d50c_69ad_4920_9320_2bd666d79a82.slice/crio-a92714f2db7ed33311bbb2d16e1f975340cd0fdf4f9118f4477814dc07417f76 WatchSource:0}: Error finding container a92714f2db7ed33311bbb2d16e1f975340cd0fdf4f9118f4477814dc07417f76: Status 404 returned error can't find the container with id a92714f2db7ed33311bbb2d16e1f975340cd0fdf4f9118f4477814dc07417f76 Jul 18 11:49:10.795255 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:10.795233 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf263af87_bce7_4fc7_8ee3_a6bf3c4d0fbf.slice/crio-d562cfe72a8d069a03def987c20c260cc2da11730e947473d0e1667e6df48380 WatchSource:0}: Error finding container d562cfe72a8d069a03def987c20c260cc2da11730e947473d0e1667e6df48380: Status 404 returned error can't find the container with id d562cfe72a8d069a03def987c20c260cc2da11730e947473d0e1667e6df48380 Jul 18 11:49:10.863855 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.863686 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 11:44:09 +0000 UTC" deadline="2028-02-20 05:38:54.956058243 +0000 UTC" Jul 18 11:49:10.863855 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.863854 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13961h49m44.092208832s" Jul 18 11:49:10.959607 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.959564 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t7ntd" event={"ID":"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f","Type":"ContainerStarted","Data":"6141f1a58d17c0eba0d717c5ffad651edaf514ac8ed2dc85ceafc7371950d321"} Jul 18 11:49:10.960429 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.960410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rqww" event={"ID":"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1","Type":"ContainerStarted","Data":"98b4563785100f6068d4270d365f7efa4fe4c63b8fff8c4f3c607d34db582e86"} Jul 18 11:49:10.961444 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.961412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerStarted","Data":"1b5f8dd8cbd8312b5a86147306764cdc0dc30f67c059bff2a5381db887d1d962"} Jul 18 11:49:10.962327 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.962309 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59zmj" event={"ID":"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5","Type":"ContainerStarted","Data":"b988aed991b9bb07b353170f47774d6272946c00eb509ea9c21bf99f5f725e22"} Jul 18 11:49:10.963221 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.963205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" event={"ID":"05c53186-396a-42a6-84c8-2d0c48c5237f","Type":"ContainerStarted","Data":"908ceb02658b1ecd8dab37ff09b9c395a76e72dcf07ebd8dc6a9b4597260f42d"} Jul 18 11:49:10.964183 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.964164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"2c838f132e6d45a7fe66887d9d816ea0ad8bb1da04df28fc31681f63deb6d5d4"} Jul 18 11:49:10.965125 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.965092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" event={"ID":"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf","Type":"ContainerStarted","Data":"d562cfe72a8d069a03def987c20c260cc2da11730e947473d0e1667e6df48380"} Jul 18 11:49:10.966216 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.966199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gmlr" event={"ID":"1ad3d50c-69ad-4920-9320-2bd666d79a82","Type":"ContainerStarted","Data":"a92714f2db7ed33311bbb2d16e1f975340cd0fdf4f9118f4477814dc07417f76"} Jul 18 11:49:10.968040 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.968023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"bbb9b55189c2ca1c136d2bbd25cb800a","Type":"ContainerStarted","Data":"8149482c35d88173e3d0948fd0748e2e2e2f7e4727bcee561e79a8a0aca309bb"} Jul 18 11:49:10.969090 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.969072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j7d7p" event={"ID":"5f205190-9537-4e79-a19b-3b796d3fda6b","Type":"ContainerStarted","Data":"e95d31f5cecbe7c1c92f5ea42881cf82e2fe44ca9e8ed705d639e04142f76d9b"} Jul 18 11:49:10.981913 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:10.981872 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" podStartSLOduration=2.981853507 podStartE2EDuration="2.981853507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:10.98123835 +0000 UTC m=+3.472010479" watchObservedRunningTime="2026-07-18 11:49:10.981853507 +0000 UTC m=+3.472625647" Jul 18 11:49:11.448504 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.448465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:11.448760 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.448545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:11.448760 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.448685 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:11.448968 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.448765 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:13.44874597 +0000 UTC m=+5.939518080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:11.448968 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.448828 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:11.448968 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.448865 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:13.448853127 +0000 UTC m=+5.939625222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:11.651232 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.651180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:11.651475 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.651337 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:11.651475 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.651359 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:11.651475 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.651372 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:11.651475 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.651429 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:13.651409539 +0000 UTC m=+6.142181641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:11.864964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.864855 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 11:44:09 +0000 UTC" deadline="2028-01-21 02:22:34.113525943 +0000 UTC" Jul 18 11:49:11.864964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.864915 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13238h33m22.24861454s" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.951543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.951671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.952068 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.952167 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.952228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:11.952422 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:11.952319 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:11.991740 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.991451 2573 generic.go:356] "Generic (PLEG): container finished" podID="3deef81c0d8c2f4458539863e90bc2e7" containerID="21edf9fea6757e8966e66058e1231f180c2a0ab69c578e816c27e72eac22a81c" exitCode=0 Jul 18 11:49:11.992820 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:11.992682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerDied","Data":"21edf9fea6757e8966e66058e1231f180c2a0ab69c578e816c27e72eac22a81c"} Jul 18 11:49:13.006194 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.006158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerStarted","Data":"e93ee3918b97425a40429c7ed8793732adf48221b3e33c915db3dee6f8793bab"} Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.468113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.468197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.468277 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.468320 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.468358 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:17.468337881 +0000 UTC m=+9.959109980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:13.468803 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.468382 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:17.468370751 +0000 UTC m=+9.959142848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:13.669953 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.669913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:13.670127 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.670077 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:13.670127 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.670099 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:13.670127 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.670113 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:13.670300 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.670173 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:17.670153382 +0000 UTC m=+10.160925479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:13.949920 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.949841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:13.949920 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.949883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:13.950132 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:13.949859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:13.950132 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.949979 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:13.950132 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.950049 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:13.950285 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:13.950135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:15.949883 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:15.949849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:15.950291 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:15.949984 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:15.950291 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:15.949849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:15.950291 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:15.950091 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:15.950291 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:15.950141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:15.950291 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:15.950203 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:17.501491 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.501452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:17.501943 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.501541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:17.501943 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.501739 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:17.501943 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.501805 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:25.501786258 +0000 UTC m=+17.992558366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:17.501943 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.501871 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:17.501943 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.501900 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:25.501890294 +0000 UTC m=+17.992662395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:17.703481 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.703433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:17.703674 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.703601 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:17.703674 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.703622 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:17.703674 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.703634 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:17.703888 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.703687 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:25.703669985 +0000 UTC m=+18.194442097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:17.950150 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.950063 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:17.950297 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.950188 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:17.950538 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.950521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:17.950645 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.950622 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:17.950694 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:17.950680 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:17.950791 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:17.950763 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:19.949267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:19.949395 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:19.949766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:19.949877 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:19.949906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:19.950013 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:19.949983 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:21.952488 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:21.952463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:21.952954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:21.952507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:21.952954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:21.952567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:21.952954 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:21.952580 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:21.952954 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:21.952658 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:21.952954 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:21.952745 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:23.949134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:23.949171 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:23.949134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:23.949263 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:23.949356 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:23.949601 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:23.949466 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:25.560540 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.560435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:25.560540 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.560523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:25.561111 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.560594 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:25.561111 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.560618 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:25.561111 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.560673 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:49:41.560651429 +0000 UTC m=+34.051423547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:25.561111 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.560693 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:49:41.560683907 +0000 UTC m=+34.051456003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:25.761543 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.761504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:25.761728 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.761669 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:25.761728 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.761691 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:25.761728 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.761701 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:25.761914 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.761773 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:49:41.761755721 +0000 UTC m=+34.252527815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:25.949285 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.949208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:25.949285 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.949250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:25.949500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:25.949218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:25.949500 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.949331 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:25.949500 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.949404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:25.949500 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:25.949484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:27.952123 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:27.952093 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:27.952547 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:27.952180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:27.952547 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:27.952183 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:27.952547 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:27.952298 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:27.952547 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:27.952335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:27.952547 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:27.952427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:29.949810 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:29.949636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:29.950231 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:29.949895 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:29.950231 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:29.949684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:29.950231 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:29.949953 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:29.950231 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:29.949656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:29.950231 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:29.949997 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:30.041190 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.041051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t7ntd" event={"ID":"d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f","Type":"ContainerStarted","Data":"121b40abdaadb74fcd125f0c4177d1c904633f66c11be3cfcdb40e48bd2e0232"} Jul 18 11:49:30.042983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.042956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rqww" event={"ID":"1866a7a6-57c5-4393-a7bf-6d3cd53e21f1","Type":"ContainerStarted","Data":"0f8006415e4cb68530a833a304f2303ea58053af5bd93bf28abda3ecb2310b6a"} Jul 18 11:49:30.044777 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.044749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerStarted","Data":"ad446e1b3d0cc797514060e55727bb33091ff4e707f327f69b98e4b5d798c8a9"} Jul 18 11:49:30.046642 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.046367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59zmj" event={"ID":"6b2d2ab6-93aa-40bb-817e-ef96789eb1b5","Type":"ContainerStarted","Data":"65904f8741aac5982f942e452ec5dc68fb75f88aa345679a3987ef80ae565a96"} Jul 18 11:49:30.048136 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.048113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" event={"ID":"05c53186-396a-42a6-84c8-2d0c48c5237f","Type":"ContainerStarted","Data":"978dfeabb49327169f6a311295711641c9bfc15505260c11188f826a76a115af"} Jul 18 11:49:30.050206 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.050181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"dca8be32856e167541437e85d772759fe8d6fb3f6172d8e075ff72e657ec19de"} Jul 18 11:49:30.050302 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.050260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"b2630ee31c05dc7635e3695378787e4eff10cbe1cbadd3393c60601302c362dc"} Jul 18 11:49:30.054034 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.053755 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" event={"ID":"f263af87-bce7-4fc7-8ee3-a6bf3c4d0fbf","Type":"ContainerStarted","Data":"574f36b5eb468e7449285d3bd34974b9add5af6fa4c7e21a1be776108cbe1005"} Jul 18 11:49:30.056399 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.056166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j7d7p" event={"ID":"5f205190-9537-4e79-a19b-3b796d3fda6b","Type":"ContainerStarted","Data":"cbf949cc7488fb2cf8239284f348b29060748a1c873393d196b4002effe3e9f7"} Jul 18 11:49:30.057334 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.057303 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t7ntd" podStartSLOduration=3.188905916 podStartE2EDuration="22.057291347s" totalImagesPullingTime="18.868385431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:30.057117348 +0000 UTC m=+22.547889465" watchObservedRunningTime="2026-07-18 11:49:30.057291347 +0000 UTC m=+22.548063464" Jul 18 11:49:30.057410 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.057393 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" podStartSLOduration=22.057389172 podStartE2EDuration="22.057389172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:13.022977291 +0000 UTC m=+5.513749408" watchObservedRunningTime="2026-07-18 11:49:30.057389172 +0000 UTC m=+22.548161286" Jul 18 11:49:30.074240 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.074189 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5rqww" podStartSLOduration=3.163742894 podStartE2EDuration="22.07417017s" totalImagesPullingTime="18.910427276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:30.074012008 +0000 UTC m=+22.564784125" watchObservedRunningTime="2026-07-18 11:49:30.07417017 +0000 UTC m=+22.564942290" Jul 18 11:49:30.112193 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.112148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-59zmj" podStartSLOduration=3.236500752 podStartE2EDuration="22.112128863s" totalImagesPullingTime="18.875628111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:30.088136561 +0000 UTC m=+22.578908679" watchObservedRunningTime="2026-07-18 11:49:30.112128863 +0000 UTC m=+22.602900978" Jul 18 11:49:30.128695 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.128647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qrlm8" podStartSLOduration=3.257799386 podStartE2EDuration="22.128632465s" totalImagesPullingTime="18.870833079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:30.128198937 +0000 UTC m=+22.618971064" watchObservedRunningTime="2026-07-18 11:49:30.128632465 +0000 UTC m=+22.619404579" Jul 18 11:49:30.145206 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.145160 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j7d7p" podStartSLOduration=3.265617485 podStartE2EDuration="22.145147173s" totalImagesPullingTime="18.879529688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:30.144998334 +0000 UTC m=+22.635770451" watchObservedRunningTime="2026-07-18 11:49:30.145147173 +0000 UTC m=+22.635919290" Jul 18 11:49:30.189319 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.188992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:30.263634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:30.263482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:31.054655 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.054620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:31.059513 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.059488 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="ad446e1b3d0cc797514060e55727bb33091ff4e707f327f69b98e4b5d798c8a9" exitCode=0 Jul 18 11:49:31.059631 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.059587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"ad446e1b3d0cc797514060e55727bb33091ff4e707f327f69b98e4b5d798c8a9"} Jul 18 11:49:31.062566 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.062549 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:49:31.062912 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.062889 2573 generic.go:356] "Generic (PLEG): container finished" podID="15e43a5d-171f-4982-b708-de1f8ef41752" containerID="dca8be32856e167541437e85d772759fe8d6fb3f6172d8e075ff72e657ec19de" exitCode=1 Jul 18 11:49:31.062974 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.062959 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerDied","Data":"dca8be32856e167541437e85d772759fe8d6fb3f6172d8e075ff72e657ec19de"} Jul 18 11:49:31.063051 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.063037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"e841e8afc78664d622c39414a6e342f80852e587c6a3efcac5d70ed82ecc1464"} Jul 18 11:49:31.063085 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.063077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"0a8493548e3bdbb37f6e50aa63ee705c3d6437bb770c08565b74e76d848976c9"} Jul 18 11:49:31.063114 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.063104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"1206d170d3ccd1a2b399b1a7eaf1867a1c9b4bacc67e5c7e0320860d83b82aaa"} Jul 18 11:49:31.063151 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.063138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"c905ea118761bb140e8c5cddb5154b4645f7f437831e50f6c330905a6162d83e"} Jul 18 11:49:31.064422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.064395 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gmlr" event={"ID":"1ad3d50c-69ad-4920-9320-2bd666d79a82","Type":"ContainerStarted","Data":"ae60a5e184fdab5799573b978eabb2a0964c553aee1d37296ccec19d692eeb2d"} Jul 18 11:49:31.092185 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.092139 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7gmlr" podStartSLOduration=5.20439114 podStartE2EDuration="24.092124526s" totalImagesPullingTime="18.887733386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:31.091589094 +0000 UTC m=+23.582361211" watchObservedRunningTime="2026-07-18 11:49:31.092124526 +0000 UTC m=+23.582896642" Jul 18 11:49:31.112476 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.112454 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 11:49:31.255290 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.255260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:31.355394 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.355364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:31.456544 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.456517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:31.901256 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.901146 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T11:49:31.112471243Z","UUID":"148a3294-9de5-4335-bec4-65f70eccd408","Handler":null,"Name":"","Endpoint":""} Jul 18 11:49:31.903330 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.903300 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 11:49:31.903473 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.903349 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.949103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.949239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:31.949249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:31.949291 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:31.949326 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:31.950593 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:31.949410 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:32.069202 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:32.069163 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" event={"ID":"05c53186-396a-42a6-84c8-2d0c48c5237f","Type":"ContainerStarted","Data":"9ec2dce74cacaf188bf9421e954352f29693f66d441a12fa3bad7b72376184b6"} Jul 18 11:49:32.494451 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:32.494360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:32.594316 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:32.594287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:32.903351 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:32.903314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:32.959824 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:32.959768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:33.077602 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.077197 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:49:33.078342 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.078231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"7aadc66af78653dd447579029bb6a9451de1e37f3ec592217085d15a1925e863"} Jul 18 11:49:33.080532 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.080496 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" event={"ID":"05c53186-396a-42a6-84c8-2d0c48c5237f","Type":"ContainerStarted","Data":"d22c62ab4cf2f7ac43122a146d91096f2308e95f6d33acfb0429cfc2c7c32e00"} Jul 18 11:49:33.097422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.097373 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs9m2" podStartSLOduration=5.188829386 podStartE2EDuration="26.097359246s" totalImagesPullingTime="20.90852986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:33.096914717 +0000 UTC m=+25.587686833" watchObservedRunningTime="2026-07-18 11:49:33.097359246 +0000 UTC m=+25.588131362" Jul 18 11:49:33.950073 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.949833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:33.950256 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.949833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:33.950256 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:33.950167 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:33.950370 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:33.950252 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:33.950370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:33.949833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:33.950370 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:33.950331 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:34.768045 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:34.768008 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:34.768640 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:34.768616 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:35.085091 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.085058 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:35.085468 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.085442 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-59zmj" Jul 18 11:49:35.140568 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.140482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:35.949574 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.949477 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:35.950411 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.949602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:35.950411 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:35.949656 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:35.950411 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:35.949701 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:35.950411 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:35.949747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:35.950411 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:35.949851 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:36.089130 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.089096 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="676fe64d03f30cf004dd3d6f03c28b0d4a5d4e75ebe5e457be75b15c6983c9c9" exitCode=0 Jul 18 11:49:36.089287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.089158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"676fe64d03f30cf004dd3d6f03c28b0d4a5d4e75ebe5e457be75b15c6983c9c9"} Jul 18 11:49:36.092489 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.092471 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:49:36.092858 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.092838 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"af2bb591aa37be94f1d8a1e65216e399301a20c5928b6eed389e4e5a362188c0"} Jul 18 11:49:36.093060 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.093037 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:36.093237 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.093220 2573 scope.go:117] "RemoveContainer" containerID="dca8be32856e167541437e85d772759fe8d6fb3f6172d8e075ff72e657ec19de" Jul 18 11:49:36.111190 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:36.110997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:37.098080 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.097983 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="b56da3f361e726c7785f4e07566be40a07e805aad0a67e4b2f039f32a7977b19" exitCode=0 Jul 18 11:49:37.098466 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.098071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"b56da3f361e726c7785f4e07566be40a07e805aad0a67e4b2f039f32a7977b19"} Jul 18 11:49:37.102356 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.102338 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:49:37.102828 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.102806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" event={"ID":"15e43a5d-171f-4982-b708-de1f8ef41752","Type":"ContainerStarted","Data":"a4a1262f0f8870093aa4de874ae03305192ae3913a3e30a90b338900cfdeb4e5"} Jul 18 11:49:37.103237 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.103149 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:37.103237 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.103234 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:37.118950 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.118914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:49:37.174391 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.174345 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" podStartSLOduration=10.234756185 podStartE2EDuration="29.174329284s" totalImagesPullingTime="18.939573099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:37.174053087 +0000 UTC m=+29.664825203" watchObservedRunningTime="2026-07-18 11:49:37.174329284 +0000 UTC m=+29.665101401" Jul 18 11:49:37.458318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.455845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfghz"] Jul 18 11:49:37.458318 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.456019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:37.458318 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:37.456146 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:37.458591 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.458442 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrb7g"] Jul 18 11:49:37.459192 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.458922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:37.459192 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:37.459069 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:37.459827 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.459804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gtc8"] Jul 18 11:49:37.459950 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:37.459907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:37.460016 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:37.459994 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:38.106838 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:38.106806 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="1d3fae4cc567463775b137c424ebdd85c71eb082bea2ef581833d5daaf928b29" exitCode=0 Jul 18 11:49:38.107356 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:38.106895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"1d3fae4cc567463775b137c424ebdd85c71eb082bea2ef581833d5daaf928b29"} Jul 18 11:49:38.949466 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:38.949441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:38.949623 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:38.949446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:38.949623 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:38.949537 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:38.949810 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:38.949638 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:39.949907 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:39.949685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:39.950353 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:39.950007 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:40.949018 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:40.948981 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:40.949210 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:40.948981 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:40.949210 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:40.949123 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:40.949326 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:40.949213 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:41.133279 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.133244 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:41.183157 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.183127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:41.592636 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.592590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:41.592830 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.592700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:41.592830 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.592772 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:41.592955 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.592830 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:41.592955 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.592839 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret podName:b665c4bf-22c3-4ea2-8c06-ab541b9d07dc nodeName:}" failed. No retries permitted until 2026-07-18 11:50:13.592819735 +0000 UTC m=+66.083591839 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret") pod "global-pull-secret-syncer-xfghz" (UID: "b665c4bf-22c3-4ea2-8c06-ab541b9d07dc") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:41.592955 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.592873 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs podName:3b052873-a80a-443c-8a2e-88bca2b4f57f nodeName:}" failed. No retries permitted until 2026-07-18 11:50:13.592862292 +0000 UTC m=+66.083634387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs") pod "network-metrics-daemon-qrb7g" (UID: "3b052873-a80a-443c-8a2e-88bca2b4f57f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:41.790268 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.790229 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:41.794031 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.794005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:41.794218 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.794199 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:41.794296 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.794227 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:41.794296 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.794239 2573 projected.go:196] Error preparing data for projected volume kube-api-access-sxwwp for pod openshift-network-diagnostics/network-check-target-6gtc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:41.794393 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.794301 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp podName:88c31a11-3a69-448b-a547-29c5b64bef7d nodeName:}" failed. No retries permitted until 2026-07-18 11:50:13.794283199 +0000 UTC m=+66.285055295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxwwp" (UniqueName: "kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp") pod "network-check-target-6gtc8" (UID: "88c31a11-3a69-448b-a547-29c5b64bef7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:41.856482 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.856443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:41.953437 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.953408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:41.953605 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:41.953530 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gtc8" podUID="88c31a11-3a69-448b-a547-29c5b64bef7d" Jul 18 11:49:41.998781 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:41.998750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:42.059006 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:42.058910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:42.949291 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:42.949214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:42.949790 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:42.949214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:42.949790 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:42.949329 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfghz" podUID="b665c4bf-22c3-4ea2-8c06-ab541b9d07dc" Jul 18 11:49:42.949790 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:42.949397 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrb7g" podUID="3b052873-a80a-443c-8a2e-88bca2b4f57f" Jul 18 11:49:43.124113 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.124083 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="1b74fd9a815e5d597465af6eeec5c57603b6591b52fd8b9fd2727a6d16fffdad" exitCode=0 Jul 18 11:49:43.124284 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.124120 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"1b74fd9a815e5d597465af6eeec5c57603b6591b52fd8b9fd2727a6d16fffdad"} Jul 18 11:49:43.820396 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.820370 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeReady" Jul 18 11:49:43.820619 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.820482 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 11:49:43.861852 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.861810 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qpc88"] Jul 18 11:49:43.882431 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.882406 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-445cn"] Jul 18 11:49:43.882575 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.882534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:43.884622 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.884601 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 11:49:43.884756 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.884605 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 11:49:43.884756 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.884680 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qxztp\"" Jul 18 11:49:43.900434 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.900416 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpc88"] Jul 18 11:49:43.900539 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.900446 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-445cn"] Jul 18 11:49:43.900539 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.900532 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:43.902782 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.902766 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbzht\"" Jul 18 11:49:43.902886 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.902766 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 11:49:43.902886 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.902767 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 11:49:43.903041 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.903010 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 11:49:43.949666 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.949639 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:49:43.951874 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.951850 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 11:49:43.951997 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.951901 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 11:49:43.952227 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.952207 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qw87\"" Jul 18 11:49:43.975791 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:43.975765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:44.012655 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtwx\" (UniqueName: \"kubernetes.io/projected/ce557499-fbb8-43d5-9527-933547932a51-kube-api-access-bbtwx\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.012791 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.012791 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce557499-fbb8-43d5-9527-933547932a51-tmp-dir\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.012905 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8nh\" (UniqueName: \"kubernetes.io/projected/83c93260-093d-4ce7-ab0d-4d238f35cc2b-kube-api-access-sn8nh\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.012905 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce557499-fbb8-43d5-9527-933547932a51-config-volume\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.012905 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.012891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c93260-093d-4ce7-ab0d-4d238f35cc2b-cert\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.113959 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.113875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.113959 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.113911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce557499-fbb8-43d5-9527-933547932a51-tmp-dir\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.114149 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.113970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8nh\" (UniqueName: \"kubernetes.io/projected/83c93260-093d-4ce7-ab0d-4d238f35cc2b-kube-api-access-sn8nh\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.114149 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.113992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce557499-fbb8-43d5-9527-933547932a51-config-volume\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.114149 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.114012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c93260-093d-4ce7-ab0d-4d238f35cc2b-cert\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.114149 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:44.114020 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:44.114149 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:44.114083 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls podName:ce557499-fbb8-43d5-9527-933547932a51 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:44.614062335 +0000 UTC m=+37.104834431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls") pod "dns-default-qpc88" (UID: "ce557499-fbb8-43d5-9527-933547932a51") : secret "dns-default-metrics-tls" not found Jul 18 11:49:44.114387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.114178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtwx\" (UniqueName: \"kubernetes.io/projected/ce557499-fbb8-43d5-9527-933547932a51-kube-api-access-bbtwx\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.114443 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.114378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce557499-fbb8-43d5-9527-933547932a51-tmp-dir\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.114626 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.114607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce557499-fbb8-43d5-9527-933547932a51-config-volume\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.118338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.118319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83c93260-093d-4ce7-ab0d-4d238f35cc2b-cert\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.122574 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.122553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtwx\" (UniqueName: \"kubernetes.io/projected/ce557499-fbb8-43d5-9527-933547932a51-kube-api-access-bbtwx\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.122665 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.122594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8nh\" (UniqueName: \"kubernetes.io/projected/83c93260-093d-4ce7-ab0d-4d238f35cc2b-kube-api-access-sn8nh\") pod \"ingress-canary-445cn\" (UID: \"83c93260-093d-4ce7-ab0d-4d238f35cc2b\") " pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.129624 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.129604 2573 generic.go:356] "Generic (PLEG): container finished" podID="afd98e56-e223-4c37-92bb-7a80e4a92e14" containerID="3a46030472050ee084b5d8ef174f37a68d27e85c4c30c7a2d274bbdf78df3cfd" exitCode=0 Jul 18 11:49:44.129732 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.129637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerDied","Data":"3a46030472050ee084b5d8ef174f37a68d27e85c4c30c7a2d274bbdf78df3cfd"} Jul 18 11:49:44.208228 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.208200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-445cn" Jul 18 11:49:44.417924 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.417678 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-445cn"] Jul 18 11:49:44.421058 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:44.421036 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c93260_093d_4ce7_ab0d_4d238f35cc2b.slice/crio-b440ff71801fa94bff5bda71edb137c8794167f3b1266ea3af2af7fe140a1fe8 WatchSource:0}: Error finding container b440ff71801fa94bff5bda71edb137c8794167f3b1266ea3af2af7fe140a1fe8: Status 404 returned error can't find the container with id b440ff71801fa94bff5bda71edb137c8794167f3b1266ea3af2af7fe140a1fe8 Jul 18 11:49:44.617808 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.617769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:44.617966 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:44.617887 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:44.617966 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:44.617942 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls podName:ce557499-fbb8-43d5-9527-933547932a51 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:45.617926903 +0000 UTC m=+38.108699020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls") pod "dns-default-qpc88" (UID: "ce557499-fbb8-43d5-9527-933547932a51") : secret "dns-default-metrics-tls" not found Jul 18 11:49:44.949568 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.949531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:49:44.949568 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.949588 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:49:44.952156 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.952132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6q4p9\"" Jul 18 11:49:44.952156 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.952148 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 11:49:44.952401 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:44.952386 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 11:49:45.136641 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.136596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcss9" event={"ID":"afd98e56-e223-4c37-92bb-7a80e4a92e14","Type":"ContainerStarted","Data":"8dadfc010fee7c2f4f4816f7437982bf1560e815effaa01e124552407e2c1fc4"} Jul 18 11:49:45.137967 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.137941 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-445cn" event={"ID":"83c93260-093d-4ce7-ab0d-4d238f35cc2b","Type":"ContainerStarted","Data":"b440ff71801fa94bff5bda71edb137c8794167f3b1266ea3af2af7fe140a1fe8"} Jul 18 11:49:45.157315 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.157245 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xcss9" podStartSLOduration=8.034412647 podStartE2EDuration="37.157225984s" totalImagesPullingTime="29.122813337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:45.155995695 +0000 UTC m=+37.646767815" watchObservedRunningTime="2026-07-18 11:49:45.157225984 +0000 UTC m=+37.647998107" Jul 18 11:49:45.280637 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.280603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:45.332932 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.332879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:45.625386 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:45.625306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:45.625531 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:45.625447 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:45.625531 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:45.625515 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls podName:ce557499-fbb8-43d5-9527-933547932a51 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:47.625497136 +0000 UTC m=+40.116269252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls") pod "dns-default-qpc88" (UID: "ce557499-fbb8-43d5-9527-933547932a51") : secret "dns-default-metrics-tls" not found Jul 18 11:49:46.679264 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.679236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:46.734966 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.734935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:46.801332 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.801299 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bvxww"] Jul 18 11:49:46.812891 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.812867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.815763 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.815744 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 11:49:46.818433 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.818417 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 11:49:46.818520 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.818450 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 11:49:46.818585 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.818544 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 11:49:46.818635 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.818596 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-29g54\"" Jul 18 11:49:46.822085 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.822063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvxww"] Jul 18 11:49:46.837172 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.837133 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:46.926138 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.926101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:46.936064 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.935861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.936064 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.936060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47k9\" (UniqueName: \"kubernetes.io/projected/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-api-access-l47k9\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.936266 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.936086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4d73aa4-e12e-4f62-96c8-b91622acbea8-crio-socket\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.936266 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.936103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4d73aa4-e12e-4f62-96c8-b91622acbea8-data-volume\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.936266 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.936125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:46.980304 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:46.980247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:47.036724 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.036677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.036896 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.036749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47k9\" (UniqueName: \"kubernetes.io/projected/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-api-access-l47k9\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.036896 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.036787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4d73aa4-e12e-4f62-96c8-b91622acbea8-crio-socket\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.036896 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.036800 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:47.036896 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.036807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4d73aa4-e12e-4f62-96c8-b91622acbea8-data-volume\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.036896 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.036867 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:47.536851439 +0000 UTC m=+40.027623555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : configmap "kube-rbac-proxy" not found Jul 18 11:49:47.037096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.036926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.037096 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.037008 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.037096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.037038 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4d73aa4-e12e-4f62-96c8-b91622acbea8-crio-socket\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.037096 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.037059 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:47.537041852 +0000 UTC m=+40.027813948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.037309 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.037294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4d73aa4-e12e-4f62-96c8-b91622acbea8-data-volume\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.046942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.046915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47k9\" (UniqueName: \"kubernetes.io/projected/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-api-access-l47k9\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.143306 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.143256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-445cn" event={"ID":"83c93260-093d-4ce7-ab0d-4d238f35cc2b","Type":"ContainerStarted","Data":"eb99b79807ae7bbf1d3fbbc2c9d6080df28cbd3ab694854cf495a0b13fe3a2ca"} Jul 18 11:49:47.158945 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.158901 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-445cn" podStartSLOduration=1.827957466 podStartE2EDuration="4.158886296s" totalImagesPullingTime="2.33092883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:47.158075374 +0000 UTC m=+39.648847491" watchObservedRunningTime="2026-07-18 11:49:47.158886296 +0000 UTC m=+39.649658415" Jul 18 11:49:47.398796 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.398763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:47.496038 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.496008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:47.538981 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.538947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.539113 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.539003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:47.539113 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.539074 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:47.539226 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.539117 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.539226 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.539136 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:48.539122442 +0000 UTC m=+41.029894536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : configmap "kube-rbac-proxy" not found Jul 18 11:49:47.539226 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.539177 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:48.539159481 +0000 UTC m=+41.029931590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.639336 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:47.639298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:47.639495 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.639464 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:47.639558 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:47.639539 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls podName:ce557499-fbb8-43d5-9527-933547932a51 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:51.639524653 +0000 UTC m=+44.130296749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls") pod "dns-default-qpc88" (UID: "ce557499-fbb8-43d5-9527-933547932a51") : secret "dns-default-metrics-tls" not found Jul 18 11:49:48.011874 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.011837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:48.036270 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.036241 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq"] Jul 18 11:49:48.070796 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.070767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq"] Jul 18 11:49:48.070925 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.070860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" Jul 18 11:49:48.073179 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.073156 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 18 11:49:48.073914 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.073893 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:48.073914 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.073906 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f8skd\"" Jul 18 11:49:48.091370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.091350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:48.149409 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.149386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:48.244472 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.244429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnzj\" (UniqueName: \"kubernetes.io/projected/87b5e01e-3574-4474-9c32-382f6409f5cf-kube-api-access-mfnzj\") pod \"migrator-7cc9c7ffd7-9gmfq\" (UID: \"87b5e01e-3574-4474-9c32-382f6409f5cf\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" Jul 18 11:49:48.276603 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.276509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:48.328640 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.328609 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:48.345189 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.345167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnzj\" (UniqueName: \"kubernetes.io/projected/87b5e01e-3574-4474-9c32-382f6409f5cf-kube-api-access-mfnzj\") pod \"migrator-7cc9c7ffd7-9gmfq\" (UID: \"87b5e01e-3574-4474-9c32-382f6409f5cf\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" Jul 18 11:49:48.353646 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.353617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnzj\" (UniqueName: \"kubernetes.io/projected/87b5e01e-3574-4474-9c32-382f6409f5cf-kube-api-access-mfnzj\") pod \"migrator-7cc9c7ffd7-9gmfq\" (UID: \"87b5e01e-3574-4474-9c32-382f6409f5cf\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" Jul 18 11:49:48.379261 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.379237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" Jul 18 11:49:48.532868 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.532792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq"] Jul 18 11:49:48.535586 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:48.535560 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b5e01e_3574_4474_9c32_382f6409f5cf.slice/crio-a3f1c8777e89251399d1ef3cd7007fc44b837609d6a1cb333d88a97e80aa6645 WatchSource:0}: Error finding container a3f1c8777e89251399d1ef3cd7007fc44b837609d6a1cb333d88a97e80aa6645: Status 404 returned error can't find the container with id a3f1c8777e89251399d1ef3cd7007fc44b837609d6a1cb333d88a97e80aa6645 Jul 18 11:49:48.546126 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.546102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:48.546214 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.546167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:48.546264 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:48.546218 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:48.546301 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:48.546271 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:50.546256264 +0000 UTC m=+43.037028360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : configmap "kube-rbac-proxy" not found Jul 18 11:49:48.546301 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:48.546273 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:48.546376 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:48.546310 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:50.546298927 +0000 UTC m=+43.037071023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:48.994407 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:48.994333 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-pdf4n"] Jul 18 11:49:49.024484 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.024452 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-pdf4n"] Jul 18 11:49:49.024956 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.024585 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.027395 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.027378 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pj8s7\"" Jul 18 11:49:49.027508 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.027400 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 18 11:49:49.027508 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.027410 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 18 11:49:49.028023 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.028006 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 18 11:49:49.028126 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.028006 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 18 11:49:49.069936 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.069910 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t7ntd_d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f/dns-node-resolver/0.log" Jul 18 11:49:49.102900 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.102875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:49.148616 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.148585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" event={"ID":"87b5e01e-3574-4474-9c32-382f6409f5cf","Type":"ContainerStarted","Data":"a3f1c8777e89251399d1ef3cd7007fc44b837609d6a1cb333d88a97e80aa6645"} Jul 18 11:49:49.149861 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.149843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-key\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.149922 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.149898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-cabundle\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.149963 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.149940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9td\" (UniqueName: \"kubernetes.io/projected/0db3d6c4-1b84-431d-b9ae-156a960b11fe-kube-api-access-lp9td\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.250503 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.250470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-key\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.250672 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.250560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-cabundle\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.250672 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.250591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9td\" (UniqueName: \"kubernetes.io/projected/0db3d6c4-1b84-431d-b9ae-156a960b11fe-kube-api-access-lp9td\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.251394 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.251374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-cabundle\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.253130 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.253106 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db3d6c4-1b84-431d-b9ae-156a960b11fe-signing-key\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.258825 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.258784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9td\" (UniqueName: \"kubernetes.io/projected/0db3d6c4-1b84-431d-b9ae-156a960b11fe-kube-api-access-lp9td\") pod \"service-ca-58598695f7-pdf4n\" (UID: \"0db3d6c4-1b84-431d-b9ae-156a960b11fe\") " pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.333068 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.333031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-pdf4n" Jul 18 11:49:49.487185 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.487158 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-pdf4n"] Jul 18 11:49:49.490734 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:49.490686 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db3d6c4_1b84_431d_b9ae_156a960b11fe.slice/crio-4e67f7bbd20ed974a06cfc844324006dd662cc82ee08850ce081d99b9807ec86 WatchSource:0}: Error finding container 4e67f7bbd20ed974a06cfc844324006dd662cc82ee08850ce081d99b9807ec86: Status 404 returned error can't find the container with id 4e67f7bbd20ed974a06cfc844324006dd662cc82ee08850ce081d99b9807ec86 Jul 18 11:49:49.870991 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.870960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j7d7p_5f205190-9537-4e79-a19b-3b796d3fda6b/node-ca/0.log" Jul 18 11:49:49.879026 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:49.878998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:50.152436 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:50.152357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-pdf4n" event={"ID":"0db3d6c4-1b84-431d-b9ae-156a960b11fe","Type":"ContainerStarted","Data":"4e67f7bbd20ed974a06cfc844324006dd662cc82ee08850ce081d99b9807ec86"} Jul 18 11:49:50.471523 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:50.471445 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-445cn_83c93260-093d-4ce7-ab0d-4d238f35cc2b/serve-healthcheck-canary/0.log" Jul 18 11:49:50.561631 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:50.561600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:50.561799 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:50.561671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:50.561799 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:50.561756 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:50.561914 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:50.561807 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:50.561914 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:50.561831 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:54.561809862 +0000 UTC m=+47.052581963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : configmap "kube-rbac-proxy" not found Jul 18 11:49:50.561914 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:50.561863 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:54.561845488 +0000 UTC m=+47.052617591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:51.156560 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:51.156515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" event={"ID":"87b5e01e-3574-4474-9c32-382f6409f5cf","Type":"ContainerStarted","Data":"10c45c5bed49a74e6576f526ae0bd57b83066696c56946eaf60f9bb8cef952c5"} Jul 18 11:49:51.156954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:51.156569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" event={"ID":"87b5e01e-3574-4474-9c32-382f6409f5cf","Type":"ContainerStarted","Data":"c77955113921ae5d2731885a85d5bb911df75da0dadfaed67e622d9f3969d1e7"} Jul 18 11:49:51.174435 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:51.174388 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-9gmfq" podStartSLOduration=1.426186548 podStartE2EDuration="3.174371529s" totalImagesPullingTime="1.748184981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:51.172776334 +0000 UTC m=+43.663548447" watchObservedRunningTime="2026-07-18 11:49:51.174371529 +0000 UTC m=+43.665143646" Jul 18 11:49:51.669823 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:51.669791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:51.669983 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:51.669902 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:51.669983 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:51.669956 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls podName:ce557499-fbb8-43d5-9527-933547932a51 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:59.66994215 +0000 UTC m=+52.160714245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls") pod "dns-default-qpc88" (UID: "ce557499-fbb8-43d5-9527-933547932a51") : secret "dns-default-metrics-tls" not found Jul 18 11:49:52.569156 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:52.569126 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.163780 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.163740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-pdf4n" event={"ID":"0db3d6c4-1b84-431d-b9ae-156a960b11fe","Type":"ContainerStarted","Data":"ecd2581bcc4c6f21d18a81d6e83f224ad24b0386fc894ec3cd2578bc64847c52"} Jul 18 11:49:53.179764 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.179698 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-pdf4n" podStartSLOduration=2.591736589 podStartE2EDuration="5.17968291s" totalImagesPullingTime="2.587946321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:53.178828654 +0000 UTC m=+45.669600772" watchObservedRunningTime="2026-07-18 11:49:53.17968291 +0000 UTC m=+45.670455026" Jul 18 11:49:53.262105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.262069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.418056 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.417953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.471830 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.471798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.577907 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.577876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.628925 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.628891 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:53.692498 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:53.692422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:54.360757 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:54.360688 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:54.592405 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:54.592371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:54.592795 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:54.592426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:49:54.592795 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:54.592510 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:54.592795 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:54.592532 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:54.592795 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:54.592560 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:02.592547317 +0000 UTC m=+55.083319412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : configmap "kube-rbac-proxy" not found Jul 18 11:49:54.592795 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:49:54.592581 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:02.592568101 +0000 UTC m=+55.083340196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:54.961609 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:54.961580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:55.157773 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:55.157733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:55.373392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:55.373361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:55.560467 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:55.560434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:55.764319 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:55.764283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:56.366742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:56.366695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:56.559177 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:56.559138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:56.956352 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:56.956318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:57.057428 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:57.057403 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:57.260371 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:57.260340 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:57.558984 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:57.558897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:57.958764 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:57.958659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:58.007154 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:58.007123 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:58.356879 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:58.356835 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:58.768135 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:58.768108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:58.958061 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:58.958021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:59.538842 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.538809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:59.591642 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.591607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:49:59.734084 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.734050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:59.736476 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.736452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce557499-fbb8-43d5-9527-933547932a51-metrics-tls\") pod \"dns-default-qpc88\" (UID: \"ce557499-fbb8-43d5-9527-933547932a51\") " pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:59.791738 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.791649 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpc88" Jul 18 11:49:59.936027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.935996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpc88"] Jul 18 11:49:59.938914 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:49:59.938887 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce557499_fbb8_43d5_9527_933547932a51.slice/crio-c63fb2dbe5a0e4ff6f84aa436fbe3eb4f9dfb92018982f28cc716c18d9987981 WatchSource:0}: Error finding container c63fb2dbe5a0e4ff6f84aa436fbe3eb4f9dfb92018982f28cc716c18d9987981: Status 404 returned error can't find the container with id c63fb2dbe5a0e4ff6f84aa436fbe3eb4f9dfb92018982f28cc716c18d9987981 Jul 18 11:49:59.960085 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:49:59.960052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:00.061566 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:00.061475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:00.163196 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:00.163149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:00.182154 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:00.182127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpc88" event={"ID":"ce557499-fbb8-43d5-9527-933547932a51","Type":"ContainerStarted","Data":"c63fb2dbe5a0e4ff6f84aa436fbe3eb4f9dfb92018982f28cc716c18d9987981"} Jul 18 11:50:00.556959 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:00.556919 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:01.158765 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:01.158733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:01.564165 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:01.564122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:01.963628 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:01.963522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:02.190001 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.189964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpc88" event={"ID":"ce557499-fbb8-43d5-9527-933547932a51","Type":"ContainerStarted","Data":"d5b04fca55ede6858a12ce5a0c9448b94018824e4cdb6c5da29030b5960c984d"} Jul 18 11:50:02.190180 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.190014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpc88" event={"ID":"ce557499-fbb8-43d5-9527-933547932a51","Type":"ContainerStarted","Data":"1b58b5a42bc60356a046a4a8a455f7ceceb66d0371808da72f0382ad40a918ba"} Jul 18 11:50:02.190180 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.190052 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qpc88" Jul 18 11:50:02.208312 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.208254 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qpc88" podStartSLOduration=17.52329848 podStartE2EDuration="19.208240158s" totalImagesPullingTime="1.684941678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:02.20690893 +0000 UTC m=+54.697681046" watchObservedRunningTime="2026-07-18 11:50:02.208240158 +0000 UTC m=+54.699012274" Jul 18 11:50:02.429583 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.429548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:02.661019 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.660978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:02.661399 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.661037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:02.661399 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:02.661114 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:50:02.661399 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:02.661180 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls podName:e4d73aa4-e12e-4f62-96c8-b91622acbea8 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:18.661163412 +0000 UTC m=+71.151935508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvxww" (UID: "e4d73aa4-e12e-4f62-96c8-b91622acbea8") : secret "insights-runtime-extractor-tls" not found Jul 18 11:50:02.661530 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.661512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4d73aa4-e12e-4f62-96c8-b91622acbea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:02.760065 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.760029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:02.964058 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:02.964028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:03.379226 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:03.379194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:03.770205 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:03.770174 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:03.960651 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:03.960620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:05.873177 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:05.873135 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:05.923946 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:05.923918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:06.181553 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:06.181447 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:06.289317 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:06.289279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:06.475229 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:06.475146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:06.863775 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:06.863736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:06.917700 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:06.917641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:07.065614 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:07.065580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:07.256294 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:07.256261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:07.657384 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:07.657282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:07.717766 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:07.717731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:07.866931 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:07.866875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:08.280102 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:08.280068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:08.546283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:08.546194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:08.686910 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:08.686870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:08.760588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:08.760549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:09.120124 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:09.120098 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfgt4" Jul 18 11:50:12.195057 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.195029 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qpc88" Jul 18 11:50:12.223617 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.223583 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh"] Jul 18 11:50:12.227520 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.227500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.230874 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.230853 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 18 11:50:12.231101 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.230966 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-znjdf\"" Jul 18 11:50:12.231210 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.230966 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 18 11:50:12.231264 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.231206 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58"] Jul 18 11:50:12.232140 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.232124 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 18 11:50:12.232438 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.232425 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 18 11:50:12.234319 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.234191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.237558 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.237542 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 18 11:50:12.237926 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.237912 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 18 11:50:12.238088 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.238076 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 18 11:50:12.238272 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.238259 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 18 11:50:12.239517 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.239499 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh"] Jul 18 11:50:12.263505 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.263470 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58"] Jul 18 11:50:12.331195 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj89n\" (UniqueName: \"kubernetes.io/projected/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-kube-api-access-vj89n\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.331387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.331387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.331387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.331387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.331387 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb667d95-3d64-4f2d-a727-f82c82d63132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.331593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblfv\" (UniqueName: \"kubernetes.io/projected/fb667d95-3d64-4f2d-a727-f82c82d63132-kube-api-access-nblfv\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.331593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.331468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.371199 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.371162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:12.432128 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432307 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb667d95-3d64-4f2d-a727-f82c82d63132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432307 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblfv\" (UniqueName: \"kubernetes.io/projected/fb667d95-3d64-4f2d-a727-f82c82d63132-kube-api-access-nblfv\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432423 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432476 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj89n\" (UniqueName: \"kubernetes.io/projected/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-kube-api-access-vj89n\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.432612 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432744 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.432883 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432860 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb667d95-3d64-4f2d-a727-f82c82d63132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.432967 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.432897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.435005 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.434979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.435293 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.435271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-ca\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.435383 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.435356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.435440 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.435407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.435518 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.435502 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb667d95-3d64-4f2d-a727-f82c82d63132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.448052 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.447994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj89n\" (UniqueName: \"kubernetes.io/projected/3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca-kube-api-access-vj89n\") pod \"managed-serviceaccount-addon-agent-8667f76b9b-ml2nh\" (UID: \"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.449355 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.449336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblfv\" (UniqueName: \"kubernetes.io/projected/fb667d95-3d64-4f2d-a727-f82c82d63132-kube-api-access-nblfv\") pod \"cluster-proxy-proxy-agent-d5fd5df-ppb58\" (UID: \"fb667d95-3d64-4f2d-a727-f82c82d63132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.462644 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.462617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:12.531497 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.531463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:12.561249 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.561223 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" Jul 18 11:50:12.569547 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.569522 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" Jul 18 11:50:12.693549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.693521 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh"] Jul 18 11:50:12.696950 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:12.696924 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b9f583d_c9cf_4c2b_8bfd_b2dc4bd80dca.slice/crio-d085c696a72936b065936696c7d7c53e58324a365c3311e69f207485e2ae78ca WatchSource:0}: Error finding container d085c696a72936b065936696c7d7c53e58324a365c3311e69f207485e2ae78ca: Status 404 returned error can't find the container with id d085c696a72936b065936696c7d7c53e58324a365c3311e69f207485e2ae78ca Jul 18 11:50:12.716006 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:12.715984 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58"] Jul 18 11:50:12.718758 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:12.718732 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb667d95_3d64_4f2d_a727_f82c82d63132.slice/crio-7e50737739a4fb8d40b24f5dc2942c50dadb536b63ae39f04f838a0f6509ba5a WatchSource:0}: Error finding container 7e50737739a4fb8d40b24f5dc2942c50dadb536b63ae39f04f838a0f6509ba5a: Status 404 returned error can't find the container with id 7e50737739a4fb8d40b24f5dc2942c50dadb536b63ae39f04f838a0f6509ba5a Jul 18 11:50:13.220820 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.220784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" event={"ID":"fb667d95-3d64-4f2d-a727-f82c82d63132","Type":"ContainerStarted","Data":"7e50737739a4fb8d40b24f5dc2942c50dadb536b63ae39f04f838a0f6509ba5a"} Jul 18 11:50:13.221688 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.221665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" event={"ID":"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca","Type":"ContainerStarted","Data":"d085c696a72936b065936696c7d7c53e58324a365c3311e69f207485e2ae78ca"} Jul 18 11:50:13.641576 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.641541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:50:13.641820 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.641621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:50:13.643990 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.643967 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 11:50:13.644116 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.643971 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 11:50:13.657405 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.657358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b052873-a80a-443c-8a2e-88bca2b4f57f-metrics-certs\") pod \"network-metrics-daemon-qrb7g\" (UID: \"3b052873-a80a-443c-8a2e-88bca2b4f57f\") " pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:50:13.657595 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.657552 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b665c4bf-22c3-4ea2-8c06-ab541b9d07dc-original-pull-secret\") pod \"global-pull-secret-syncer-xfghz\" (UID: \"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc\") " pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:50:13.761445 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.761408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfghz" Jul 18 11:50:13.770124 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.770094 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6q4p9\"" Jul 18 11:50:13.777937 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.777913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrb7g" Jul 18 11:50:13.843607 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.843508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:50:13.845932 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.845892 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 11:50:13.856141 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.856105 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 11:50:13.880631 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.880603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwwp\" (UniqueName: \"kubernetes.io/projected/88c31a11-3a69-448b-a547-29c5b64bef7d-kube-api-access-sxwwp\") pod \"network-check-target-6gtc8\" (UID: \"88c31a11-3a69-448b-a547-29c5b64bef7d\") " pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:50:13.960260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.960232 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qw87\"" Jul 18 11:50:13.968400 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.968356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:50:13.979571 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:13.979199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfghz"] Jul 18 11:50:13.985992 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:13.985962 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb665c4bf_22c3_4ea2_8c06_ab541b9d07dc.slice/crio-12b538fd212d38ac89a9e3f01c8b864cc204567e636e938edc06d02e76220b6d WatchSource:0}: Error finding container 12b538fd212d38ac89a9e3f01c8b864cc204567e636e938edc06d02e76220b6d: Status 404 returned error can't find the container with id 12b538fd212d38ac89a9e3f01c8b864cc204567e636e938edc06d02e76220b6d Jul 18 11:50:14.001000 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.000973 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrb7g"] Jul 18 11:50:14.008215 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:14.008186 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b052873_a80a_443c_8a2e_88bca2b4f57f.slice/crio-8b81748e604ee0231f4b4df4c2e08588076b48390ba8c32385b88455fa7d901d WatchSource:0}: Error finding container 8b81748e604ee0231f4b4df4c2e08588076b48390ba8c32385b88455fa7d901d: Status 404 returned error can't find the container with id 8b81748e604ee0231f4b4df4c2e08588076b48390ba8c32385b88455fa7d901d Jul 18 11:50:14.177696 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.177582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gtc8"] Jul 18 11:50:14.180800 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:14.180767 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c31a11_3a69_448b_a547_29c5b64bef7d.slice/crio-360c0d61da6153b57f15d72906b1744bc42ac7dd9a7ec1c4fdcd2428d9fcb750 WatchSource:0}: Error finding container 360c0d61da6153b57f15d72906b1744bc42ac7dd9a7ec1c4fdcd2428d9fcb750: Status 404 returned error can't find the container with id 360c0d61da6153b57f15d72906b1744bc42ac7dd9a7ec1c4fdcd2428d9fcb750 Jul 18 11:50:14.227520 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.227477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrb7g" event={"ID":"3b052873-a80a-443c-8a2e-88bca2b4f57f","Type":"ContainerStarted","Data":"8b81748e604ee0231f4b4df4c2e08588076b48390ba8c32385b88455fa7d901d"} Jul 18 11:50:14.229065 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.229007 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gtc8" event={"ID":"88c31a11-3a69-448b-a547-29c5b64bef7d","Type":"ContainerStarted","Data":"360c0d61da6153b57f15d72906b1744bc42ac7dd9a7ec1c4fdcd2428d9fcb750"} Jul 18 11:50:14.230752 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.230684 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfghz" event={"ID":"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc","Type":"ContainerStarted","Data":"12b538fd212d38ac89a9e3f01c8b864cc204567e636e938edc06d02e76220b6d"} Jul 18 11:50:14.292732 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.292671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:14.352173 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:14.352138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:17.582303 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:17.582234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:17.682669 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:17.682575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:17.768821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:17.768627 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:17.959565 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:17.959476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:18.061105 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.061052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:18.177419 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.177383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:18.249451 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.249412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrb7g" event={"ID":"3b052873-a80a-443c-8a2e-88bca2b4f57f","Type":"ContainerStarted","Data":"4452869fd9327874dd2ec23ac4bbe650a399c0aba7fd9bf7ffd677ed2300021a"} Jul 18 11:50:18.251151 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.251117 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" event={"ID":"fb667d95-3d64-4f2d-a727-f82c82d63132","Type":"ContainerStarted","Data":"3914045b8c9d28e5a94b7c8a7e85998aa3dca78c380980c9c7667fd28b5ca18b"} Jul 18 11:50:18.252664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.252636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" event={"ID":"3b9f583d-c9cf-4c2b-8bfd-b2dc4bd80dca","Type":"ContainerStarted","Data":"12b9a9209c61e26e5ed90126f0d382cb5af50c72e6283dab8ba277a093cbd027"} Jul 18 11:50:18.269809 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.269746 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8667f76b9b-ml2nh" podStartSLOduration=3.328429671 podStartE2EDuration="6.269726691s" totalImagesPullingTime="2.94129702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:18.268599847 +0000 UTC m=+70.759371955" watchObservedRunningTime="2026-07-18 11:50:18.269726691 +0000 UTC m=+70.760498803" Jul 18 11:50:18.683221 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.683136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:18.685949 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.685922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4d73aa4-e12e-4f62-96c8-b91622acbea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvxww\" (UID: \"e4d73aa4-e12e-4f62-96c8-b91622acbea8\") " pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:18.696311 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.696262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:18.789263 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.789226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:18.932542 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.932508 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-29g54\"" Jul 18 11:50:18.941429 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:18.941362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvxww" Jul 18 11:50:20.338492 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:20.338468 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvxww"] Jul 18 11:50:20.340902 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:20.340873 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d73aa4_e12e_4f62_96c8_b91622acbea8.slice/crio-f9cf096480844c6e4d66e302b31120dc50bef10d900f756bdc772b323fb51ac1 WatchSource:0}: Error finding container f9cf096480844c6e4d66e302b31120dc50bef10d900f756bdc772b323fb51ac1: Status 404 returned error can't find the container with id f9cf096480844c6e4d66e302b31120dc50bef10d900f756bdc772b323fb51ac1 Jul 18 11:50:20.935913 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:20.935755 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.037748 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.037694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.265923 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.265888 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrb7g" event={"ID":"3b052873-a80a-443c-8a2e-88bca2b4f57f","Type":"ContainerStarted","Data":"8a6107973a6665942668794d16c744980e238ec1fe1dcdb73792ac70e145b9f3"} Jul 18 11:50:21.267625 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.267600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gtc8" event={"ID":"88c31a11-3a69-448b-a547-29c5b64bef7d","Type":"ContainerStarted","Data":"c3b72ee3858d1661654e9007f8e26e9a80f3c2e989640b931057f4e03f5a708f"} Jul 18 11:50:21.267783 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.267738 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:50:21.269182 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.269160 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfghz" event={"ID":"b665c4bf-22c3-4ea2-8c06-ab541b9d07dc","Type":"ContainerStarted","Data":"3f695e0f47f78295d52dc663523e785313bd27bfc6a9d6d9b1b9f1db74d8437f"} Jul 18 11:50:21.270482 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.270456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvxww" event={"ID":"e4d73aa4-e12e-4f62-96c8-b91622acbea8","Type":"ContainerStarted","Data":"f9cf096480844c6e4d66e302b31120dc50bef10d900f756bdc772b323fb51ac1"} Jul 18 11:50:21.272767 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.272746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" event={"ID":"fb667d95-3d64-4f2d-a727-f82c82d63132","Type":"ContainerStarted","Data":"6323ac0693e3df5a9356010cb705a1c0cea2feb18ac94f8d1a0db8a9b2cf33c0"} Jul 18 11:50:21.272865 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.272778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" event={"ID":"fb667d95-3d64-4f2d-a727-f82c82d63132","Type":"ContainerStarted","Data":"27a34f72cdc2d124f69d58af25a08372bd1c060073e19a9689df698f6ce3596d"} Jul 18 11:50:21.281466 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.281405 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qrb7g" podStartSLOduration=69.680641302 podStartE2EDuration="1m13.281389272s" totalImagesPullingTime="3.60074797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:21.280790373 +0000 UTC m=+73.771562490" watchObservedRunningTime="2026-07-18 11:50:21.281389272 +0000 UTC m=+73.772161390" Jul 18 11:50:21.294505 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.294459 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xfghz" podStartSLOduration=67.104183861 podStartE2EDuration="1m13.294446331s" totalImagesPullingTime="6.19026247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:21.293838173 +0000 UTC m=+73.784610291" watchObservedRunningTime="2026-07-18 11:50:21.294446331 +0000 UTC m=+73.785218484" Jul 18 11:50:21.307991 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.307950 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6gtc8" podStartSLOduration=67.302020992 podStartE2EDuration="1m13.307934703s" totalImagesPullingTime="6.005913711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:21.307267787 +0000 UTC m=+73.798039905" watchObservedRunningTime="2026-07-18 11:50:21.307934703 +0000 UTC m=+73.798706820" Jul 18 11:50:21.324066 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.324018 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5fd5df-ppb58" podStartSLOduration=4.411564774 podStartE2EDuration="9.324007618s" totalImagesPullingTime="4.912442844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:21.323209756 +0000 UTC m=+73.813981900" watchObservedRunningTime="2026-07-18 11:50:21.324007618 +0000 UTC m=+73.814779735" Jul 18 11:50:21.410258 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.410221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.509378 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.509349 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.761404 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.761369 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.834153 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.834117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:21.913256 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:21.913221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:24.576951 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:24.576913 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:24.677440 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:24.677407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:25.101348 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:25.101312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:25.163770 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:25.163740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:25.286450 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:25.286405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvxww" event={"ID":"e4d73aa4-e12e-4f62-96c8-b91622acbea8","Type":"ContainerStarted","Data":"a88717213db0599bdf3f8e18943fe93a56399637d86eb64a21ffa5b27881b10b"} Jul 18 11:50:25.660400 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:25.660364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:27.917588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:27.917544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.170998 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.170868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.273240 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.273105 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bbwrw"] Jul 18 11:50:28.285946 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.285915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.288561 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.288369 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 11:50:28.288897 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.288863 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 11:50:28.288897 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.288892 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-grnx4\"" Jul 18 11:50:28.289061 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.288865 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 11:50:28.289452 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.289435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 11:50:28.289549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.289528 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 11:50:28.289549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.289526 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 11:50:28.301695 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.301667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.352246 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.351923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352246 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.351995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-metrics-client-ca\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352246 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352246 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvgh\" (UniqueName: \"kubernetes.io/projected/8444571b-e2f0-45d1-a0bf-2d29a060e216-kube-api-access-rnvgh\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-sys\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-wtmp\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.352942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-root\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.353370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.352885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.353370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.353231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-textfile\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455007 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.454923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-metrics-client-ca\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455007 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.454972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvgh\" (UniqueName: \"kubernetes.io/projected/8444571b-e2f0-45d1-a0bf-2d29a060e216-kube-api-access-rnvgh\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-sys\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-wtmp\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-root\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-textfile\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-sys\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455789 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-root\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455789 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:28.455450 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 11:50:28.455789 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:28.455507 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls podName:8444571b-e2f0-45d1-a0bf-2d29a060e216 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:28.955487804 +0000 UTC m=+81.446259901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls") pod "node-exporter-bbwrw" (UID: "8444571b-e2f0-45d1-a0bf-2d29a060e216") : secret "node-exporter-tls" not found Jul 18 11:50:28.455789 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.455789 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.455765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-wtmp\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.456041 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.456021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-textfile\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.456172 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.456153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8444571b-e2f0-45d1-a0bf-2d29a060e216-metrics-client-ca\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.458133 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.458079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.478829 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.478772 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvgh\" (UniqueName: \"kubernetes.io/projected/8444571b-e2f0-45d1-a0bf-2d29a060e216-kube-api-access-rnvgh\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.512270 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.512192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.604923 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.604877 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.895929 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.895899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:28.959271 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:28.959178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:28.959661 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:28.959294 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 11:50:28.959661 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:50:28.959360 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls podName:8444571b-e2f0-45d1-a0bf-2d29a060e216 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:29.959345203 +0000 UTC m=+82.450117297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls") pod "node-exporter-bbwrw" (UID: "8444571b-e2f0-45d1-a0bf-2d29a060e216") : secret "node-exporter-tls" not found Jul 18 11:50:29.184898 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.184867 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:29.278276 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.278243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:29.304757 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.304658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvxww" event={"ID":"e4d73aa4-e12e-4f62-96c8-b91622acbea8","Type":"ContainerStarted","Data":"17a8e7052d071372935e52bca55f69e13d50ccf302ba225b82bcc4c3f08ac73c"} Jul 18 11:50:29.346168 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.346128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:29.596041 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.595954 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:29.699508 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.699421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:29.966476 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.966398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:29.969040 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:29.969015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8444571b-e2f0-45d1-a0bf-2d29a060e216-node-exporter-tls\") pod \"node-exporter-bbwrw\" (UID: \"8444571b-e2f0-45d1-a0bf-2d29a060e216\") " pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:30.099769 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:30.099735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbwrw" Jul 18 11:50:30.112554 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:30.112522 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8444571b_e2f0_45d1_a0bf_2d29a060e216.slice/crio-b4cdef788c65eda875d6cfe5a835e70d09a059295ca7b8a0029db1f40b0bd291 WatchSource:0}: Error finding container b4cdef788c65eda875d6cfe5a835e70d09a059295ca7b8a0029db1f40b0bd291: Status 404 returned error can't find the container with id b4cdef788c65eda875d6cfe5a835e70d09a059295ca7b8a0029db1f40b0bd291 Jul 18 11:50:30.308988 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:30.308954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbwrw" event={"ID":"8444571b-e2f0-45d1-a0bf-2d29a060e216","Type":"ContainerStarted","Data":"b4cdef788c65eda875d6cfe5a835e70d09a059295ca7b8a0029db1f40b0bd291"} Jul 18 11:50:30.417775 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:30.417735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:30.519300 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:30.519257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:31.100495 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:31.100463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:31.194365 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:31.194281 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:31.688756 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:31.688698 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:31.791688 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:31.791652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:32.709625 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:32.709535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:32.914401 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:32.914370 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:32.970993 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:32.970887 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:33.233603 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.233500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:33.321783 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.321749 2573 generic.go:356] "Generic (PLEG): container finished" podID="8444571b-e2f0-45d1-a0bf-2d29a060e216" containerID="81a3bdba8e777318e10b15e472b9624bdd6feac9cd55419662549ce0691f97d6" exitCode=0 Jul 18 11:50:33.321959 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.321828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbwrw" event={"ID":"8444571b-e2f0-45d1-a0bf-2d29a060e216","Type":"ContainerDied","Data":"81a3bdba8e777318e10b15e472b9624bdd6feac9cd55419662549ce0691f97d6"} Jul 18 11:50:33.324027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.323992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvxww" event={"ID":"e4d73aa4-e12e-4f62-96c8-b91622acbea8","Type":"ContainerStarted","Data":"0c1fd0738405c4928376103d9019fd050f89454e9b90acdec9fd33e2b819da66"} Jul 18 11:50:33.364083 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.364042 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bvxww" podStartSLOduration=35.377131322 podStartE2EDuration="47.364025287s" totalImagesPullingTime="11.986893965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:33.363101083 +0000 UTC m=+85.853873216" watchObservedRunningTime="2026-07-18 11:50:33.364025287 +0000 UTC m=+85.854797425" Jul 18 11:50:33.652377 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.652341 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:33.753075 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:33.753038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:34.329643 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.329607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbwrw" event={"ID":"8444571b-e2f0-45d1-a0bf-2d29a060e216","Type":"ContainerStarted","Data":"bcef2045f5a4067be190edef7408831d86051a8d902370b1227a5592c648701a"} Jul 18 11:50:34.329834 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.329662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbwrw" event={"ID":"8444571b-e2f0-45d1-a0bf-2d29a060e216","Type":"ContainerStarted","Data":"cd98f42e39accf55e2bb5eae2b2d9bd64b156443c5b2c47453146e79968acbd7"} Jul 18 11:50:34.352919 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.352872 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bbwrw" podStartSLOduration=3.936102962 podStartE2EDuration="6.352857216s" totalImagesPullingTime="2.341845815s" totalInitContainerRuntime="74.908439ms" isStatefulPod=true podCreationTimestamp="2026-07-18 11:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:34.351471281 +0000 UTC m=+86.842243398" watchObservedRunningTime="2026-07-18 11:50:34.352857216 +0000 UTC m=+86.843629333" Jul 18 11:50:34.378086 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.378057 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 18 11:50:34.381421 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.381405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.384059 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.384039 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 18 11:50:34.384980 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.384959 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 18 11:50:34.384980 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.384963 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 18 11:50:34.385139 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.385033 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 18 11:50:34.385139 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.385111 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 18 11:50:34.386288 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386272 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 18 11:50:34.386381 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386291 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 18 11:50:34.386381 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 18 11:50:34.386561 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386547 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 18 11:50:34.386633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386586 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-prq2t\"" Jul 18 11:50:34.386896 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386882 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 18 11:50:34.386981 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.386959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 18 11:50:34.387039 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.387017 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 18 11:50:34.387090 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.387077 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2j0j18mmoi0lm\"" Jul 18 11:50:34.387480 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.387461 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 18 11:50:34.389281 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.389264 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 18 11:50:34.396752 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.393437 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 18 11:50:34.400536 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.400512 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 18 11:50:34.403479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.403454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:34.504420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdvg\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-kube-api-access-5rdvg\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504851 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.504983 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.504952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.529405 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.529371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:34.605621 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605621 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605621 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605899 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605899 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605899 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.605899 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.605749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606242 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606242 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdvg\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-kube-api-access-5rdvg\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.606633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.606605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.607106 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.607080 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.607482 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.607457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.607876 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.607857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.609995 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.609514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.609995 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.609553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.609995 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.609951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.610192 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.610063 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf7da949-0300-47cd-af4f-dfe579cfe1f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.610245 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.610218 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.610909 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.610883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.611014 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.610889 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.611159 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.611107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf7da949-0300-47cd-af4f-dfe579cfe1f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.611809 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.611778 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.612108 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.612086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.612178 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.612142 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.612500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.612481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.612650 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.612633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.612812 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.612793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf7da949-0300-47cd-af4f-dfe579cfe1f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.617641 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.617616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdvg\" (UniqueName: \"kubernetes.io/projected/bf7da949-0300-47cd-af4f-dfe579cfe1f6-kube-api-access-5rdvg\") pod \"prometheus-k8s-0\" (UID: \"bf7da949-0300-47cd-af4f-dfe579cfe1f6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.691277 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.691241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:34.830481 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:34.830456 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 18 11:50:34.832503 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:50:34.832478 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7da949_0300_47cd_af4f_dfe579cfe1f6.slice/crio-ba74f3dffd728754f11efe20dc82ddebefb6a18773d5a90ae9854f09a46bc12c WatchSource:0}: Error finding container ba74f3dffd728754f11efe20dc82ddebefb6a18773d5a90ae9854f09a46bc12c: Status 404 returned error can't find the container with id ba74f3dffd728754f11efe20dc82ddebefb6a18773d5a90ae9854f09a46bc12c Jul 18 11:50:35.108158 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:35.108118 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:35.208942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:35.208905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:35.311749 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:35.311687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:35.335616 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:35.335568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"ba74f3dffd728754f11efe20dc82ddebefb6a18773d5a90ae9854f09a46bc12c"} Jul 18 11:50:36.340050 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:36.340014 2573 generic.go:356] "Generic (PLEG): container finished" podID="bf7da949-0300-47cd-af4f-dfe579cfe1f6" containerID="0f5fa0fc3ec6e1d4daa589a192dc31b119f637446192bcd4ed74fff3eb7c1fcb" exitCode=0 Jul 18 11:50:36.340410 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:36.340103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerDied","Data":"0f5fa0fc3ec6e1d4daa589a192dc31b119f637446192bcd4ed74fff3eb7c1fcb"} Jul 18 11:50:40.358687 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:40.358647 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"88a19f2faa75c346aaa767549a1a5812a83841df486981fd2395669fb0e7bafc"} Jul 18 11:50:40.359089 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:40.358744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"1ab6b3714855a6baf407b90dcba4f4ab97d8fb2e750c77d9cafe824e7c1b25de"} Jul 18 11:50:41.133620 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:41.133578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:41.183879 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:41.183839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:41.365554 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:41.365523 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"37945a833ba09a6b820ddd792a8c1ae6fd4aad8f34b977ada5e2696808f7dea9"} Jul 18 11:50:42.372172 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:42.372137 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"1a6a0417ff5f1d78776d7cdb7a7f42cee8956702b252609a31d8f242ac84c33d"} Jul 18 11:50:42.372559 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:42.372207 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"2a0b7a777bf13e39fbb12259e362693cdb24be0ec81085c6c92fd3c5a4941e91"} Jul 18 11:50:42.372559 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:42.372235 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf7da949-0300-47cd-af4f-dfe579cfe1f6","Type":"ContainerStarted","Data":"62cd6525edff5eab6e19283790024282169ac8c19f3647df7317b61897e1bd03"} Jul 18 11:50:42.400948 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:42.400847 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.627713271 podStartE2EDuration="8.400826221s" totalImagesPullingTime="5.75458997s" totalInitContainerRuntime="18.52298ms" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:42.399212061 +0000 UTC m=+94.889984178" watchObservedRunningTime="2026-07-18 11:50:42.400826221 +0000 UTC m=+94.891598339" Jul 18 11:50:44.184094 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:44.184050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:44.245013 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:44.244974 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:44.303093 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:44.303055 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:44.692395 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:44.692346 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:50:45.120920 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:45.120884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:45.221759 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:45.221701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:52.278121 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:52.278091 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6gtc8" Jul 18 11:50:56.144882 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:56.144842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:50:56.244753 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:50:56.244722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:03.195379 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:03.195342 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:03.296992 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:03.296956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:03.361825 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:03.361788 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:05.333314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:05.333273 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:05.434626 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:05.434590 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:25.651469 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:25.651434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:34.692336 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:34.692304 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:51:34.711680 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:34.711655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:51:35.575117 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:35.575091 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 18 11:51:47.381847 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:47.381805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:47.498314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:47.498275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:56.371259 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:56.371223 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:56.510263 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:56.510227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:56.610423 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:56.610390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:57.111635 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:57.111600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:51:57.212106 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:51:57.212072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:02.572149 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:02.572066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:02.672577 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:02.672550 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:14.545020 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:14.544980 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:14.645314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:14.645279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:17.548025 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:17.547989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:17.599525 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:17.599501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:17.651435 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:17.651412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:26.141145 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:26.141107 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:26.241522 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:26.241492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:26.298743 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:26.298692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:48.143645 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:48.143610 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:48.244819 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:48.244792 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:49.628369 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:49.628328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:52:49.718089 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:52:49.718054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:11.351575 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:11.351542 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:12.130283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:12.130245 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:14.633422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:14.633390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:14.934393 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:14.934317 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:15.034955 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:15.034924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:15.944346 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:15.944310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:16.044395 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:16.044365 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:18.327149 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:18.327119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:18.756600 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:18.756572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:21.327371 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:21.327335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:21.390386 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:21.390347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:25.155224 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:25.155191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:25.216351 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:25.216316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:25.970274 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:25.970241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:28.396137 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:28.396048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:28.496887 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:28.496857 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:34.470977 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:34.470937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:34.527602 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:34.527573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:36.395230 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:36.395198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:46.635569 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:46.635536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:46.713184 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:46.713154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:51.955842 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:51.955807 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:52.074845 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.074811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:52.571603 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.571565 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:52.742131 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.742098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:52.862762 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.862665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:52.911083 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.911052 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7"] Jul 18 11:53:52.914242 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.914224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:52.916688 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.916666 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 11:53:52.916832 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.916755 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m7xcv\"" Jul 18 11:53:52.916832 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.916820 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 11:53:52.916938 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.916913 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 11:53:52.917033 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.917015 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 11:53:52.926654 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:52.926635 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7"] Jul 18 11:53:53.037785 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.037759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwffj\" (UniqueName: \"kubernetes.io/projected/094f6b24-9ca2-4d76-b3d4-50ca9051e262-kube-api-access-nwffj\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.038187 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.037793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.038187 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.037811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.038187 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.037900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.038187 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.037928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.050831 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.050805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:53.138499 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.138499 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.138499 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.138752 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.138752 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwffj\" (UniqueName: \"kubernetes.io/projected/094f6b24-9ca2-4d76-b3d4-50ca9051e262-kube-api-access-nwffj\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.138922 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.138898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.140990 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.140962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/094f6b24-9ca2-4d76-b3d4-50ca9051e262-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.141187 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.141173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.141233 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.141202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/094f6b24-9ca2-4d76-b3d4-50ca9051e262-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.150820 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.150791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwffj\" (UniqueName: \"kubernetes.io/projected/094f6b24-9ca2-4d76-b3d4-50ca9051e262-kube-api-access-nwffj\") pod \"opendatahub-operator-controller-manager-fbb59dc99-p2ms7\" (UID: \"094f6b24-9ca2-4d76-b3d4-50ca9051e262\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.151828 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.151803 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:53.224761 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.224733 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:53.251502 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.251477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:53.354340 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.354301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7"] Jul 18 11:53:53.357211 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.357187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:53.357668 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:53:53.357646 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094f6b24_9ca2_4d76_b3d4_50ca9051e262.slice/crio-72a50fd70e955f766e7f02f881c1661bbe43beb351ed369f9d6f821d0ad1991e WatchSource:0}: Error finding container 72a50fd70e955f766e7f02f881c1661bbe43beb351ed369f9d6f821d0ad1991e: Status 404 returned error can't find the container with id 72a50fd70e955f766e7f02f881c1661bbe43beb351ed369f9d6f821d0ad1991e Jul 18 11:53:53.452224 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:53.452146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:53:54.022731 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:54.022685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" event={"ID":"094f6b24-9ca2-4d76-b3d4-50ca9051e262","Type":"ContainerStarted","Data":"72a50fd70e955f766e7f02f881c1661bbe43beb351ed369f9d6f821d0ad1991e"} Jul 18 11:53:57.034998 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:57.034965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" event={"ID":"094f6b24-9ca2-4d76-b3d4-50ca9051e262","Type":"ContainerStarted","Data":"063dc43161fcfa2c6c2f5f7ac5a51c19388a46a7a2bfc6a18c09d8201fe08d4e"} Jul 18 11:53:58.040040 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:58.040008 2573 generic.go:356] "Generic (PLEG): container finished" podID="094f6b24-9ca2-4d76-b3d4-50ca9051e262" containerID="063dc43161fcfa2c6c2f5f7ac5a51c19388a46a7a2bfc6a18c09d8201fe08d4e" exitCode=0 Jul 18 11:53:58.040491 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:58.040096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" event={"ID":"094f6b24-9ca2-4d76-b3d4-50ca9051e262","Type":"ContainerDied","Data":"063dc43161fcfa2c6c2f5f7ac5a51c19388a46a7a2bfc6a18c09d8201fe08d4e"} Jul 18 11:53:59.045438 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:59.045403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" event={"ID":"094f6b24-9ca2-4d76-b3d4-50ca9051e262","Type":"ContainerStarted","Data":"b61192b821ebdb8a1e2187480de00a93774db5cd6a36b19c16ad19f84a77eec3"} Jul 18 11:53:59.045878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:59.045532 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:53:59.066765 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:53:59.066701 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" podStartSLOduration=2.745925435 podStartE2EDuration="7.06668599s" totalImagesPullingTime="3.318864917s" totalInitContainerRuntime="1.001895638s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:53:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:53:59.065466379 +0000 UTC m=+291.556238508" watchObservedRunningTime="2026-07-18 11:53:59.06668599 +0000 UTC m=+291.557458106" Jul 18 11:54:07.835420 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:07.835388 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:54:07.837856 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:07.837832 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:54:07.843686 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:07.843658 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 11:54:10.051717 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:10.051686 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-p2ms7" Jul 18 11:54:11.737591 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:11.737561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:12.595248 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.595217 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4ww5d"] Jul 18 11:54:12.597227 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.597211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.600721 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.600684 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k7mmt\"" Jul 18 11:54:12.601014 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.600997 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 18 11:54:12.611415 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.611394 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4ww5d"] Jul 18 11:54:12.785895 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.785864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.786260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.785933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwn7\" (UniqueName: \"kubernetes.io/projected/c4243ffa-19fc-479c-9d31-39a703a4ab0a-kube-api-access-8hwn7\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.793586 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.793557 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:12.886283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.886222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwn7\" (UniqueName: \"kubernetes.io/projected/c4243ffa-19fc-479c-9d31-39a703a4ab0a-kube-api-access-8hwn7\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.886283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.886273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.886420 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:12.886378 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 18 11:54:12.886454 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:12.886441 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert podName:c4243ffa-19fc-479c-9d31-39a703a4ab0a nodeName:}" failed. No retries permitted until 2026-07-18 11:54:13.386421987 +0000 UTC m=+305.877194103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert") pod "odh-model-controller-7998c67dc4-4ww5d" (UID: "c4243ffa-19fc-479c-9d31-39a703a4ab0a") : secret "odh-model-controller-webhook-cert" not found Jul 18 11:54:12.900026 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.900002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwn7\" (UniqueName: \"kubernetes.io/projected/c4243ffa-19fc-479c-9d31-39a703a4ab0a-kube-api-access-8hwn7\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:12.903485 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.903457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:12.995667 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:12.995635 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:13.211488 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.211420 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2wxw"] Jul 18 11:54:13.213479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.213466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.215766 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.215746 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 18 11:54:13.224256 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.224235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2wxw"] Jul 18 11:54:13.227391 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.227371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:13.355720 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.355671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:13.389640 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.389617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3204df0-de56-4810-9dbf-43b1c86826bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.389729 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.389668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxd7\" (UniqueName: \"kubernetes.io/projected/e3204df0-de56-4810-9dbf-43b1c86826bd-kube-api-access-jvxd7\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.389795 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.389777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:13.392178 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.392152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4243ffa-19fc-479c-9d31-39a703a4ab0a-cert\") pod \"odh-model-controller-7998c67dc4-4ww5d\" (UID: \"c4243ffa-19fc-479c-9d31-39a703a4ab0a\") " pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:13.490449 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.490384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3204df0-de56-4810-9dbf-43b1c86826bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.490449 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.490427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxd7\" (UniqueName: \"kubernetes.io/projected/e3204df0-de56-4810-9dbf-43b1c86826bd-kube-api-access-jvxd7\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.492993 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.492968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3204df0-de56-4810-9dbf-43b1c86826bd-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.498287 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.498268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxd7\" (UniqueName: \"kubernetes.io/projected/e3204df0-de56-4810-9dbf-43b1c86826bd-kube-api-access-jvxd7\") pod \"model-serving-api-54d89bd5fc-k2wxw\" (UID: \"e3204df0-de56-4810-9dbf-43b1c86826bd\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.507165 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.507148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:13.524055 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.524034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:13.643008 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.642981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4ww5d"] Jul 18 11:54:13.645004 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:13.644980 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4243ffa_19fc_479c_9d31_39a703a4ab0a.slice/crio-52660e318e43e64cef67d5efcb1e2ac30afa4cbcc0907b160db09b6098b9dd0e WatchSource:0}: Error finding container 52660e318e43e64cef67d5efcb1e2ac30afa4cbcc0907b160db09b6098b9dd0e: Status 404 returned error can't find the container with id 52660e318e43e64cef67d5efcb1e2ac30afa4cbcc0907b160db09b6098b9dd0e Jul 18 11:54:13.646370 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.646352 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:54:13.662312 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:13.662292 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2wxw"] Jul 18 11:54:13.664972 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:13.664951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3204df0_de56_4810_9dbf_43b1c86826bd.slice/crio-3cbdf1d182a585533c1de78fb09511c7fbb573bacd6f4463cf1de06c244eb380 WatchSource:0}: Error finding container 3cbdf1d182a585533c1de78fb09511c7fbb573bacd6f4463cf1de06c244eb380: Status 404 returned error can't find the container with id 3cbdf1d182a585533c1de78fb09511c7fbb573bacd6f4463cf1de06c244eb380 Jul 18 11:54:14.101391 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:14.101358 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" event={"ID":"e3204df0-de56-4810-9dbf-43b1c86826bd","Type":"ContainerStarted","Data":"3cbdf1d182a585533c1de78fb09511c7fbb573bacd6f4463cf1de06c244eb380"} Jul 18 11:54:14.102321 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:14.102299 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" event={"ID":"c4243ffa-19fc-479c-9d31-39a703a4ab0a","Type":"ContainerStarted","Data":"52660e318e43e64cef67d5efcb1e2ac30afa4cbcc0907b160db09b6098b9dd0e"} Jul 18 11:54:17.116893 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.116856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" event={"ID":"c4243ffa-19fc-479c-9d31-39a703a4ab0a","Type":"ContainerStarted","Data":"3456f00b15abf0b66f6dbec12b52d4d1c46ee0207bbd60123d63710813068f59"} Jul 18 11:54:17.117369 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.116929 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:17.118145 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.118113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" event={"ID":"e3204df0-de56-4810-9dbf-43b1c86826bd","Type":"ContainerStarted","Data":"671f708e5c7f87e1e09eeab5d626fe9503fd06b47f767c69e16aa9d9ec43a65d"} Jul 18 11:54:17.118294 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.118255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:17.132813 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.132767 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" podStartSLOduration=2.3766850919999998 podStartE2EDuration="5.132751588s" totalImagesPullingTime="2.756066496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:17.131786931 +0000 UTC m=+309.622559045" watchObservedRunningTime="2026-07-18 11:54:17.132751588 +0000 UTC m=+309.623523704" Jul 18 11:54:17.146907 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:17.146871 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" podStartSLOduration=1.45591722 podStartE2EDuration="4.146855119s" totalImagesPullingTime="2.690937899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:17.14642078 +0000 UTC m=+309.637192915" watchObservedRunningTime="2026-07-18 11:54:17.146855119 +0000 UTC m=+309.637627259" Jul 18 11:54:21.291119 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.291072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:21.338396 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.338363 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx"] Jul 18 11:54:21.341680 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.341659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.344682 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.344660 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 18 11:54:21.345270 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.345254 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-czh6n\"" Jul 18 11:54:21.351453 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.351429 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn"] Jul 18 11:54:21.353739 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.353550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bd8\" (UniqueName: \"kubernetes.io/projected/2f375ae5-c14b-4d08-8641-209764156fac-kube-api-access-94bd8\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.353739 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.353626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.354736 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.354696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.355408 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.355388 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx"] Jul 18 11:54:21.359622 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.357959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6ktvr\"" Jul 18 11:54:21.359622 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.358157 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 18 11:54:21.381377 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.366014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn"] Jul 18 11:54:21.381377 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.370577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.463334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.463400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bd8\" (UniqueName: \"kubernetes.io/projected/2f375ae5-c14b-4d08-8641-209764156fac-kube-api-access-94bd8\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.463444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqgk\" (UniqueName: \"kubernetes.io/projected/4bd352bc-fadc-40f7-b772-ed0be93c08db-kube-api-access-nsqgk\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.463488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:21.463605 2573 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 18 11:54:21.464961 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:21.463678 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert podName:2f375ae5-c14b-4d08-8641-209764156fac nodeName:}" failed. No retries permitted until 2026-07-18 11:54:21.963646647 +0000 UTC m=+314.454418742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert") pod "kserve-controller-manager-d6f4b65b8-2dgqx" (UID: "2f375ae5-c14b-4d08-8641-209764156fac") : secret "kserve-webhook-server-cert" not found Jul 18 11:54:21.474474 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.474434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bd8\" (UniqueName: \"kubernetes.io/projected/2f375ae5-c14b-4d08-8641-209764156fac-kube-api-access-94bd8\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.563935 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.563858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.563935 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.563925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqgk\" (UniqueName: \"kubernetes.io/projected/4bd352bc-fadc-40f7-b772-ed0be93c08db-kube-api-access-nsqgk\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.564113 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:21.564013 2573 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 18 11:54:21.564113 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:21.564089 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert podName:4bd352bc-fadc-40f7-b772-ed0be93c08db nodeName:}" failed. No retries permitted until 2026-07-18 11:54:22.064068825 +0000 UTC m=+314.554840920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert") pod "llmisvc-controller-manager-777c6bd7d5-5v9zn" (UID: "4bd352bc-fadc-40f7-b772-ed0be93c08db") : secret "llmisvc-webhook-server-cert" not found Jul 18 11:54:21.587454 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.587422 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqgk\" (UniqueName: \"kubernetes.io/projected/4bd352bc-fadc-40f7-b772-ed0be93c08db-kube-api-access-nsqgk\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:21.729824 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.729782 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:21.830192 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.830096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:21.965554 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.965522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:21.967915 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:21.967891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f375ae5-c14b-4d08-8641-209764156fac-cert\") pod \"kserve-controller-manager-d6f4b65b8-2dgqx\" (UID: \"2f375ae5-c14b-4d08-8641-209764156fac\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:22.066355 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.066324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:22.069162 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.069136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd352bc-fadc-40f7-b772-ed0be93c08db-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5v9zn\" (UID: \"4bd352bc-fadc-40f7-b772-ed0be93c08db\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:22.252818 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.252792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:22.268404 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.268383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:22.388617 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.388594 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx"] Jul 18 11:54:22.392108 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:22.392074 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f375ae5_c14b_4d08_8641_209764156fac.slice/crio-875ab64d2adca3d71b95570262d5e91a34d273b705b5f1a1213888aca29d8e9f WatchSource:0}: Error finding container 875ab64d2adca3d71b95570262d5e91a34d273b705b5f1a1213888aca29d8e9f: Status 404 returned error can't find the container with id 875ab64d2adca3d71b95570262d5e91a34d273b705b5f1a1213888aca29d8e9f Jul 18 11:54:22.415957 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.415932 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn"] Jul 18 11:54:22.418357 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:22.418334 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bd352bc_fadc_40f7_b772_ed0be93c08db.slice/crio-9a79db7959e1140d59c913f4ec7cf900f931d6f1fdabfdec828e8ff9eafc483a WatchSource:0}: Error finding container 9a79db7959e1140d59c913f4ec7cf900f931d6f1fdabfdec828e8ff9eafc483a: Status 404 returned error can't find the container with id 9a79db7959e1140d59c913f4ec7cf900f931d6f1fdabfdec828e8ff9eafc483a Jul 18 11:54:22.609601 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.609512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:22.689028 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.688996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:22.787618 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.787587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:22.912156 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.912077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:22.947834 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.947805 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w"] Jul 18 11:54:22.951307 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.951284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:22.957891 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.957868 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 11:54:22.958240 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.958225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 11:54:22.958549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.958447 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 18 11:54:22.958549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.958468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2n9js\"" Jul 18 11:54:22.958858 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.958837 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 18 11:54:22.965664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.965645 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w"] Jul 18 11:54:22.972818 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.972798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tmp\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:22.972934 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.972857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:22.973436 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:22.973196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txv8d\" (UniqueName: \"kubernetes.io/projected/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-kube-api-access-txv8d\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.074093 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.074045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txv8d\" (UniqueName: \"kubernetes.io/projected/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-kube-api-access-txv8d\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.074260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.074127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tmp\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.074260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.074165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.074374 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:23.074283 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 18 11:54:23.074374 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:54:23.074339 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs podName:6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f nodeName:}" failed. No retries permitted until 2026-07-18 11:54:23.574319936 +0000 UTC m=+316.065092032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs") pod "kube-auth-proxy-677848b8b7-gfq4w" (UID: "6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f") : secret "kube-auth-proxy-tls" not found Jul 18 11:54:23.076766 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.076745 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tmp\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.088620 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.088566 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txv8d\" (UniqueName: \"kubernetes.io/projected/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-kube-api-access-txv8d\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.143063 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.143027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" event={"ID":"2f375ae5-c14b-4d08-8641-209764156fac","Type":"ContainerStarted","Data":"875ab64d2adca3d71b95570262d5e91a34d273b705b5f1a1213888aca29d8e9f"} Jul 18 11:54:23.144434 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.144396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" event={"ID":"4bd352bc-fadc-40f7-b772-ed0be93c08db","Type":"ContainerStarted","Data":"9a79db7959e1140d59c913f4ec7cf900f931d6f1fdabfdec828e8ff9eafc483a"} Jul 18 11:54:23.309416 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.309357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:23.578917 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.578829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.581447 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.581419 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f-tls-certs\") pod \"kube-auth-proxy-677848b8b7-gfq4w\" (UID: \"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f\") " pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:23.862646 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:23.862561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" Jul 18 11:54:24.049979 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.049952 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w"] Jul 18 11:54:24.561615 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:24.561575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9a2e6b_cc3e_479d_8e4c_683722a7ad7f.slice/crio-7954e9e1af2d90c7408011f99062cab8310bf9843d458469395e095574b81abc WatchSource:0}: Error finding container 7954e9e1af2d90c7408011f99062cab8310bf9843d458469395e095574b81abc: Status 404 returned error can't find the container with id 7954e9e1af2d90c7408011f99062cab8310bf9843d458469395e095574b81abc Jul 18 11:54:24.573473 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.573448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:24.580437 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.580415 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs"] Jul 18 11:54:24.583354 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.583338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.585776 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.585748 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 18 11:54:24.585878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.585856 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 18 11:54:24.585878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.585860 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 18 11:54:24.585987 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.585932 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 11:54:24.586162 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586146 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dsm2k\"" Jul 18 11:54:24.586231 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e0d20f-a7c3-434f-a3ca-f5989228c73a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586231 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586231 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68szw\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-kube-api-access-68szw\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586358 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586358 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586426 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.586426 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.586375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.599629 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.599609 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs"] Jul 18 11:54:24.669299 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.669267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:24.687557 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687680 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687680 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e0d20f-a7c3-434f-a3ca-f5989228c73a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.687821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.687802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68szw\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-kube-api-access-68szw\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.688615 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.688570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.690588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.690450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.690588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.690475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.690588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.690526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56e0d20f-a7c3-434f-a3ca-f5989228c73a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.690849 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.690679 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e0d20f-a7c3-434f-a3ca-f5989228c73a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.712666 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.712623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68szw\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-kube-api-access-68szw\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.716499 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.716472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e0d20f-a7c3-434f-a3ca-f5989228c73a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-flsqs\" (UID: \"56e0d20f-a7c3-434f-a3ca-f5989228c73a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:24.892410 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:24.892324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:25.154821 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.154730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" event={"ID":"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f","Type":"ContainerStarted","Data":"7954e9e1af2d90c7408011f99062cab8310bf9843d458469395e095574b81abc"} Jul 18 11:54:25.156602 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.156569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" event={"ID":"2f375ae5-c14b-4d08-8641-209764156fac","Type":"ContainerStarted","Data":"d0ea8ade536588a301a39f31a5746dcb3b70f73fc091b44f270253350c50ede9"} Jul 18 11:54:25.156890 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.156869 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:25.173161 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.173110 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" podStartSLOduration=1.956026698 podStartE2EDuration="4.173089977s" totalImagesPullingTime="2.217063279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:25.171568125 +0000 UTC m=+317.662340254" watchObservedRunningTime="2026-07-18 11:54:25.173089977 +0000 UTC m=+317.663862093" Jul 18 11:54:25.399182 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.399154 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs"] Jul 18 11:54:25.403490 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:54:25.403457 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e0d20f_a7c3_434f_a3ca_f5989228c73a.slice/crio-ed37cfa755000af48c72964aa4bfa4a9aebb542b7821b86fdde929885ec6ce73 WatchSource:0}: Error finding container ed37cfa755000af48c72964aa4bfa4a9aebb542b7821b86fdde929885ec6ce73: Status 404 returned error can't find the container with id ed37cfa755000af48c72964aa4bfa4a9aebb542b7821b86fdde929885ec6ce73 Jul 18 11:54:25.785103 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:25.785071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:26.164362 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:26.164271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" event={"ID":"4bd352bc-fadc-40f7-b772-ed0be93c08db","Type":"ContainerStarted","Data":"098cccb4914b82630fd84e584061792b7fa590509fb0cd4a731b2e3921760c8a"} Jul 18 11:54:26.164513 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:26.164428 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:54:26.168206 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:26.168178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" event={"ID":"56e0d20f-a7c3-434f-a3ca-f5989228c73a","Type":"ContainerStarted","Data":"ed37cfa755000af48c72964aa4bfa4a9aebb542b7821b86fdde929885ec6ce73"} Jul 18 11:54:26.185224 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:26.184996 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" podStartSLOduration=2.277908366 podStartE2EDuration="5.184912303s" totalImagesPullingTime="2.907003937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:26.182053981 +0000 UTC m=+318.672826099" watchObservedRunningTime="2026-07-18 11:54:26.184912303 +0000 UTC m=+318.675684421" Jul 18 11:54:28.099885 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.099830 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:54:28.100206 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.099902 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:54:28.124662 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.124638 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4ww5d" Jul 18 11:54:28.127362 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.127344 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-k2wxw" Jul 18 11:54:28.179281 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.179249 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" event={"ID":"6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f","Type":"ContainerStarted","Data":"b152656bbe5a72acf192bfada28bb809d8691d5331daefc5da136d79b02ab1b4"} Jul 18 11:54:28.181278 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:28.181251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" event={"ID":"56e0d20f-a7c3-434f-a3ca-f5989228c73a","Type":"ContainerStarted","Data":"1c975cf57b74748b076fd9e0bd9259c05ca8f263ff4d39b2fc92a5882ca5fa6d"} Jul 18 11:54:29.185449 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.185421 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:29.187047 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.187021 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-flsqs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 11:54:29.187191 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.187076 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" podUID="56e0d20f-a7c3-434f-a3ca-f5989228c73a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 11:54:29.203570 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.203519 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-677848b8b7-gfq4w" podStartSLOduration=3.669599441 podStartE2EDuration="7.203501887s" totalImagesPullingTime="3.533902446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:29.202780126 +0000 UTC m=+321.693552247" watchObservedRunningTime="2026-07-18 11:54:29.203501887 +0000 UTC m=+321.694274006" Jul 18 11:54:29.223627 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.223175 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" podStartSLOduration=2.530622083 podStartE2EDuration="5.223160427s" totalImagesPullingTime="2.692538344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:29.222825718 +0000 UTC m=+321.713597835" watchObservedRunningTime="2026-07-18 11:54:29.223160427 +0000 UTC m=+321.713932545" Jul 18 11:54:29.326975 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.326938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:29.429541 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.429506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:29.980024 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:29.979993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:30.068159 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:30.068128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:30.168350 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:30.168316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:30.190361 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:30.190342 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-flsqs" Jul 18 11:54:30.719016 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:30.718982 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:54:56.174052 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:56.174027 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2dgqx" Jul 18 11:54:57.175335 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:54:57.175308 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5v9zn" Jul 18 11:55:13.145040 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:13.144952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:13.238976 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:13.238945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:13.343081 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:13.343045 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:14.068986 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:14.068955 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:14.269624 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:14.269592 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:55.216276 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:55.216238 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:55.809790 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:55.809754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:59.200167 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:59.200127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:55:59.300106 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:55:59.300076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:00.635186 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:00.635143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:02.664178 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:02.664138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:02.766720 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:02.766672 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:03.045500 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:03.045467 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:03.154439 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:03.154407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:03.632794 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:03.632759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:04.454217 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:04.454186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:05.190391 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:05.190357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:05.290853 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:05.290814 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:06.611996 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:06.611956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:07.087430 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:07.087390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:10.272308 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.272276 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:10.824557 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.824525 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb"] Jul 18 11:56:10.826657 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.826642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:10.829314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.829293 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 11:56:10.829314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.829304 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qvmch\"" Jul 18 11:56:10.829314 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.829304 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 11:56:10.844349 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.844327 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb"] Jul 18 11:56:10.896894 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.896863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdd6p\" (UniqueName: \"kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:10.897029 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.896910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:10.997804 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.997764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:10.998006 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.997985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdd6p\" (UniqueName: \"kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:10.998195 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:10.998168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:11.010058 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:11.010028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdd6p\" (UniqueName: \"kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:11.136551 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:11.136468 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:11.267816 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:11.267698 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb"] Jul 18 11:56:11.270663 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:56:11.270631 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775ba957_5acc_4459_a336_07c74e1c67a5.slice/crio-bda9caac190114b8e97fd86e3c4084f50fcee8d092d145c1fc8c688a08706ca0 WatchSource:0}: Error finding container bda9caac190114b8e97fd86e3c4084f50fcee8d092d145c1fc8c688a08706ca0: Status 404 returned error can't find the container with id bda9caac190114b8e97fd86e3c4084f50fcee8d092d145c1fc8c688a08706ca0 Jul 18 11:56:11.602181 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:11.602142 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" event={"ID":"775ba957-5acc-4459-a336-07c74e1c67a5","Type":"ContainerStarted","Data":"bda9caac190114b8e97fd86e3c4084f50fcee8d092d145c1fc8c688a08706ca0"} Jul 18 11:56:12.232592 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:12.232551 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:12.333769 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:12.333728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:12.641102 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:12.641057 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:12.735319 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:12.735282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:16.025211 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:16.025169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:16.078827 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:16.078798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:16.628589 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:16.628558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" event={"ID":"775ba957-5acc-4459-a336-07c74e1c67a5","Type":"ContainerStarted","Data":"51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699"} Jul 18 11:56:16.628794 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:16.628690 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:16.649344 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:16.649296 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" podStartSLOduration=1.722438961 podStartE2EDuration="6.64928396s" totalImagesPullingTime="4.926844999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:16.648067027 +0000 UTC m=+429.138839163" watchObservedRunningTime="2026-07-18 11:56:16.64928396 +0000 UTC m=+429.140056076" Jul 18 11:56:18.332218 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:18.332185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:18.429313 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:18.429276 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:18.957473 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:18.957439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:19.125682 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:19.125650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:19.838468 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:19.838435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:19.936425 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:19.936389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:23.355664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:23.355632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:23.455798 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:23.455767 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:27.634293 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:27.634269 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:28.442766 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.442686 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq"] Jul 18 11:56:28.447240 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.447224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.462345 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.462319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq"] Jul 18 11:56:28.537048 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.537020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.537154 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.537073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9hh\" (UniqueName: \"kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.638312 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.638284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.638616 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.638346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9hh\" (UniqueName: \"kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.638736 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.638693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.650698 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.650677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9hh\" (UniqueName: \"kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.756074 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.756058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:28.886394 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:28.886363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq"] Jul 18 11:56:28.889576 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:56:28.889550 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda067d308_fbab_4cb6_8e1a_092457516778.slice/crio-56f393d221ccc05c467783950fe216be9bbcbaba9dcca25f3137d5c032876e48 WatchSource:0}: Error finding container 56f393d221ccc05c467783950fe216be9bbcbaba9dcca25f3137d5c032876e48: Status 404 returned error can't find the container with id 56f393d221ccc05c467783950fe216be9bbcbaba9dcca25f3137d5c032876e48 Jul 18 11:56:29.234118 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.234045 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq"] Jul 18 11:56:29.237947 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.237914 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq"] Jul 18 11:56:29.246267 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.246245 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb"] Jul 18 11:56:29.246445 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.246423 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" containerName="manager" containerID="cri-o://51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699" gracePeriod=2 Jul 18 11:56:29.252907 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.252883 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb"] Jul 18 11:56:29.258737 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.258697 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:29.259023 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259009 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a067d308-fbab-4cb6-8e1a-092457516778" containerName="manager" Jul 18 11:56:29.259096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259025 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a067d308-fbab-4cb6-8e1a-092457516778" containerName="manager" Jul 18 11:56:29.259096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259041 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" containerName="manager" Jul 18 11:56:29.259096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259047 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" containerName="manager" Jul 18 11:56:29.259096 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259093 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a067d308-fbab-4cb6-8e1a-092457516778" containerName="manager" Jul 18 11:56:29.259296 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.259099 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" containerName="manager" Jul 18 11:56:29.259948 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.259907 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"775ba957-5acc-4459-a336-07c74e1c67a5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.263278 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.263257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.265078 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.265054 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.274450 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.274430 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:29.277751 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.277729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.278722 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.278683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:29.294677 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.294653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:29.320361 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.320325 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.343204 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.343177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.343322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.343213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85px\" (UniqueName: \"kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.343322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.343283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcqq\" (UniqueName: \"kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.343322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.343308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.444263 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcqq\" (UniqueName: \"kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.444398 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.444398 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.444398 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85px\" (UniqueName: \"kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.444737 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.444841 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.444700 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.454133 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.454111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85px\" (UniqueName: \"kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lrhml\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.455524 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.455504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcqq\" (UniqueName: \"kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-849jq\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.475386 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.475367 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.477387 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.477361 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.544846 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.544818 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume\") pod \"775ba957-5acc-4459-a336-07c74e1c67a5\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " Jul 18 11:56:29.544958 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.544887 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdd6p\" (UniqueName: \"kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p\") pod \"775ba957-5acc-4459-a336-07c74e1c67a5\" (UID: \"775ba957-5acc-4459-a336-07c74e1c67a5\") " Jul 18 11:56:29.545243 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.545220 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "775ba957-5acc-4459-a336-07c74e1c67a5" (UID: "775ba957-5acc-4459-a336-07c74e1c67a5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 11:56:29.547202 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.547184 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p" (OuterVolumeSpecName: "kube-api-access-hdd6p") pod "775ba957-5acc-4459-a336-07c74e1c67a5" (UID: "775ba957-5acc-4459-a336-07c74e1c67a5"). InnerVolumeSpecName "kube-api-access-hdd6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:56:29.613901 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.613876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:29.619891 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.619876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:29.646003 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.645972 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdd6p\" (UniqueName: \"kubernetes.io/projected/775ba957-5acc-4459-a336-07c74e1c67a5-kube-api-access-hdd6p\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:29.646003 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.645995 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/775ba957-5acc-4459-a336-07c74e1c67a5-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:29.689877 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.689770 2573 generic.go:356] "Generic (PLEG): container finished" podID="775ba957-5acc-4459-a336-07c74e1c67a5" containerID="51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699" exitCode=0 Jul 18 11:56:29.689877 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.689872 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.690029 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.689868 2573 scope.go:117] "RemoveContainer" containerID="51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699" Jul 18 11:56:29.692315 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.692283 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.693168 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.693144 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" container="manager" Jul 18 11:56:29.695938 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.695808 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="a067d308-fbab-4cb6-8e1a-092457516778" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:29.697636 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.697614 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.702863 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.702778 2573 scope.go:117] "RemoveContainer" containerID="51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699" Jul 18 11:56:29.704034 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.703782 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699\": container with ID starting with 51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699 not found: ID does not exist" containerID="51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699" Jul 18 11:56:29.704034 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.703832 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699"} err="failed to get container status \"51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699\": rpc error: code = NotFound desc = could not find container \"51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699\": container with ID starting with 51cc592a65c50e72284b63204112638c3c578d7d2ed9c73ae42d975ad7010699 not found: ID does not exist" Jul 18 11:56:29.705997 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.705966 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="a067d308-fbab-4cb6-8e1a-092457516778" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:29.710563 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:29.710512 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zbwxb\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbwxb" Jul 18 11:56:29.766651 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.766622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:29.770805 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:56:29.770782 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c4284_7ca0_4e34_9d75_759a35d52c2a.slice/crio-73bd7c96c9c0f26195c2ccaa6e8f1d8fe39ca6474afce73330cf2a9187ea35d4 WatchSource:0}: Error finding container 73bd7c96c9c0f26195c2ccaa6e8f1d8fe39ca6474afce73330cf2a9187ea35d4: Status 404 returned error can't find the container with id 73bd7c96c9c0f26195c2ccaa6e8f1d8fe39ca6474afce73330cf2a9187ea35d4 Jul 18 11:56:29.789606 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.789587 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:29.791933 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:56:29.791907 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fab89b_c8c0_4273_b2dd_2fe76add906b.slice/crio-a3a9eadbce1c3fd847288ca83c6c295a88713a0efed69effa97219af10dd9ab2 WatchSource:0}: Error finding container a3a9eadbce1c3fd847288ca83c6c295a88713a0efed69effa97219af10dd9ab2: Status 404 returned error can't find the container with id a3a9eadbce1c3fd847288ca83c6c295a88713a0efed69effa97219af10dd9ab2 Jul 18 11:56:29.955052 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:29.955023 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775ba957-5acc-4459-a336-07c74e1c67a5" path="/var/lib/kubelet/pods/775ba957-5acc-4459-a336-07c74e1c67a5/volumes" Jul 18 11:56:30.699386 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.699348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" event={"ID":"cb0c4284-7ca0-4e34-9d75-759a35d52c2a","Type":"ContainerStarted","Data":"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65"} Jul 18 11:56:30.699861 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.699409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" event={"ID":"cb0c4284-7ca0-4e34-9d75-759a35d52c2a","Type":"ContainerStarted","Data":"73bd7c96c9c0f26195c2ccaa6e8f1d8fe39ca6474afce73330cf2a9187ea35d4"} Jul 18 11:56:30.699861 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.699441 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:30.701891 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.701860 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" podUID="a067d308-fbab-4cb6-8e1a-092457516778" containerName="manager" containerID="cri-o://a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267" gracePeriod=2 Jul 18 11:56:30.703361 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.703339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" event={"ID":"f6fab89b-c8c0-4273-b2dd-2fe76add906b","Type":"ContainerStarted","Data":"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036"} Jul 18 11:56:30.703473 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.703384 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" event={"ID":"f6fab89b-c8c0-4273-b2dd-2fe76add906b","Type":"ContainerStarted","Data":"a3a9eadbce1c3fd847288ca83c6c295a88713a0efed69effa97219af10dd9ab2"} Jul 18 11:56:30.703473 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.703411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:30.719695 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.719648 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" podStartSLOduration=1.71963205 podStartE2EDuration="1.71963205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:30.717818622 +0000 UTC m=+443.208590741" watchObservedRunningTime="2026-07-18 11:56:30.71963205 +0000 UTC m=+443.210404166" Jul 18 11:56:30.741854 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.741808 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" podStartSLOduration=1.741796115 podStartE2EDuration="1.741796115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:30.740201385 +0000 UTC m=+443.230973503" watchObservedRunningTime="2026-07-18 11:56:30.741796115 +0000 UTC m=+443.232568231" Jul 18 11:56:30.941284 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:30.941263 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:30.943557 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:30.943534 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="a067d308-fbab-4cb6-8e1a-092457516778" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:31.059025 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.058982 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume\") pod \"a067d308-fbab-4cb6-8e1a-092457516778\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " Jul 18 11:56:31.059190 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.059045 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s9hh\" (UniqueName: \"kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh\") pod \"a067d308-fbab-4cb6-8e1a-092457516778\" (UID: \"a067d308-fbab-4cb6-8e1a-092457516778\") " Jul 18 11:56:31.059329 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.059303 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a067d308-fbab-4cb6-8e1a-092457516778" (UID: "a067d308-fbab-4cb6-8e1a-092457516778"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 11:56:31.061227 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.061198 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh" (OuterVolumeSpecName: "kube-api-access-4s9hh") pod "a067d308-fbab-4cb6-8e1a-092457516778" (UID: "a067d308-fbab-4cb6-8e1a-092457516778"). InnerVolumeSpecName "kube-api-access-4s9hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:56:31.160479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.160446 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9hh\" (UniqueName: \"kubernetes.io/projected/a067d308-fbab-4cb6-8e1a-092457516778-kube-api-access-4s9hh\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:31.160479 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.160477 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a067d308-fbab-4cb6-8e1a-092457516778-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:31.395949 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.395866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:31.494065 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.494032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:31.710392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.710316 2573 generic.go:356] "Generic (PLEG): container finished" podID="a067d308-fbab-4cb6-8e1a-092457516778" containerID="a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267" exitCode=0 Jul 18 11:56:31.710392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.710359 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:31.710392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.710375 2573 scope.go:117] "RemoveContainer" containerID="a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267" Jul 18 11:56:31.712649 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:31.712627 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="a067d308-fbab-4cb6-8e1a-092457516778" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:31.719032 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.719019 2573 scope.go:117] "RemoveContainer" containerID="a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267" Jul 18 11:56:31.719296 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:31.719281 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267\": container with ID starting with a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267 not found: ID does not exist" containerID="a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267" Jul 18 11:56:31.719338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.719304 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267"} err="failed to get container status \"a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267\": rpc error: code = NotFound desc = could not find container \"a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267\": container with ID starting with a276d482297aa0580d7dd128e45d5c8da1ed09e85426584f9779eb39f922d267 not found: ID does not exist" Jul 18 11:56:31.719959 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:31.719942 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qjcdq\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="a067d308-fbab-4cb6-8e1a-092457516778" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qjcdq" Jul 18 11:56:31.955031 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.955002 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a067d308-fbab-4cb6-8e1a-092457516778" path="/var/lib/kubelet/pods/a067d308-fbab-4cb6-8e1a-092457516778/volumes" Jul 18 11:56:31.994587 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:31.994560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:32.095217 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:32.095190 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:34.990321 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:34.990284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:35.091135 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:35.091095 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:39.597260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:39.597219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:39.698075 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:39.698039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:41.712816 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:41.712784 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:41.713173 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:41.712859 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:41.785599 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:41.785572 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:41.785787 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:41.785765 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" podUID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" containerName="manager" containerID="cri-o://bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036" gracePeriod=10 Jul 18 11:56:42.021492 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.021466 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:42.146286 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.146254 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgcqq\" (UniqueName: \"kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq\") pod \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " Jul 18 11:56:42.146470 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.146336 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume\") pod \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\" (UID: \"f6fab89b-c8c0-4273-b2dd-2fe76add906b\") " Jul 18 11:56:42.146786 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.146759 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f6fab89b-c8c0-4273-b2dd-2fe76add906b" (UID: "f6fab89b-c8c0-4273-b2dd-2fe76add906b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 11:56:42.148626 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.148609 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq" (OuterVolumeSpecName: "kube-api-access-dgcqq") pod "f6fab89b-c8c0-4273-b2dd-2fe76add906b" (UID: "f6fab89b-c8c0-4273-b2dd-2fe76add906b"). InnerVolumeSpecName "kube-api-access-dgcqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:56:42.247157 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.247082 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcqq\" (UniqueName: \"kubernetes.io/projected/f6fab89b-c8c0-4273-b2dd-2fe76add906b-kube-api-access-dgcqq\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:42.247157 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.247105 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6fab89b-c8c0-4273-b2dd-2fe76add906b-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:42.761256 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.761223 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" containerID="bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036" exitCode=0 Jul 18 11:56:42.761664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.761269 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" Jul 18 11:56:42.761664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.761304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" event={"ID":"f6fab89b-c8c0-4273-b2dd-2fe76add906b","Type":"ContainerDied","Data":"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036"} Jul 18 11:56:42.761664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.761356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq" event={"ID":"f6fab89b-c8c0-4273-b2dd-2fe76add906b","Type":"ContainerDied","Data":"a3a9eadbce1c3fd847288ca83c6c295a88713a0efed69effa97219af10dd9ab2"} Jul 18 11:56:42.761664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.761378 2573 scope.go:117] "RemoveContainer" containerID="bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036" Jul 18 11:56:42.770792 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.770776 2573 scope.go:117] "RemoveContainer" containerID="bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036" Jul 18 11:56:42.771026 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:42.771010 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036\": container with ID starting with bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036 not found: ID does not exist" containerID="bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036" Jul 18 11:56:42.771076 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.771034 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036"} err="failed to get container status \"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036\": rpc error: code = NotFound desc = could not find container \"bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036\": container with ID starting with bbd8eb565e747e4ed349e4b266ec3d5291756069ec977585d44fd1a04eba6036 not found: ID does not exist" Jul 18 11:56:42.785789 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.785765 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:42.790330 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:42.790310 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-849jq"] Jul 18 11:56:43.086249 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:43.086173 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:43.186315 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:43.186282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:43.954408 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:43.954380 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" path="/var/lib/kubelet/pods/f6fab89b-c8c0-4273-b2dd-2fe76add906b/volumes" Jul 18 11:56:58.096913 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:58.096874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:58.513086 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:58.513051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:56:58.793053 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:58.792976 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:58.793221 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:58.793196 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" podUID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" containerName="manager" containerID="cri-o://db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65" gracePeriod=10 Jul 18 11:56:59.051024 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.050969 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:59.071815 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.071757 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m85px\" (UniqueName: \"kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px\") pod \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " Jul 18 11:56:59.072035 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.071898 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume\") pod \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\" (UID: \"cb0c4284-7ca0-4e34-9d75-759a35d52c2a\") " Jul 18 11:56:59.072328 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.072279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb0c4284-7ca0-4e34-9d75-759a35d52c2a" (UID: "cb0c4284-7ca0-4e34-9d75-759a35d52c2a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 11:56:59.074549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.074524 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px" (OuterVolumeSpecName: "kube-api-access-m85px") pod "cb0c4284-7ca0-4e34-9d75-759a35d52c2a" (UID: "cb0c4284-7ca0-4e34-9d75-759a35d52c2a"). InnerVolumeSpecName "kube-api-access-m85px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:56:59.173239 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.173220 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:59.173239 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.173240 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85px\" (UniqueName: \"kubernetes.io/projected/cb0c4284-7ca0-4e34-9d75-759a35d52c2a-kube-api-access-m85px\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:56:59.839482 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.839448 2573 generic.go:356] "Generic (PLEG): container finished" podID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" containerID="db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65" exitCode=0 Jul 18 11:56:59.839634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.839491 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" Jul 18 11:56:59.839634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.839523 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" event={"ID":"cb0c4284-7ca0-4e34-9d75-759a35d52c2a","Type":"ContainerDied","Data":"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65"} Jul 18 11:56:59.839634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.839584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml" event={"ID":"cb0c4284-7ca0-4e34-9d75-759a35d52c2a","Type":"ContainerDied","Data":"73bd7c96c9c0f26195c2ccaa6e8f1d8fe39ca6474afce73330cf2a9187ea35d4"} Jul 18 11:56:59.839634 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.839614 2573 scope.go:117] "RemoveContainer" containerID="db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65" Jul 18 11:56:59.848766 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.848749 2573 scope.go:117] "RemoveContainer" containerID="db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65" Jul 18 11:56:59.849019 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:56:59.849002 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65\": container with ID starting with db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65 not found: ID does not exist" containerID="db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65" Jul 18 11:56:59.849067 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.849029 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65"} err="failed to get container status \"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65\": rpc error: code = NotFound desc = could not find container \"db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65\": container with ID starting with db5aa6bb7d5b222f1e2c5f087d1dad03d14a46da5d252070f685322d00a1ba65 not found: ID does not exist" Jul 18 11:56:59.863561 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.863538 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:59.867949 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.867927 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lrhml"] Jul 18 11:56:59.955069 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:56:59.955035 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" path="/var/lib/kubelet/pods/cb0c4284-7ca0-4e34-9d75-759a35d52c2a/volumes" Jul 18 11:57:06.182074 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:06.182039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:06.284061 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:06.284024 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.067467 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.067418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.122050 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.122022 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.212878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.212837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.265646 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.265615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.815073 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.815038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:07.868976 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:07.868952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.032965 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.032927 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.141839 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.141803 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.243014 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.242988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.453172 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453100 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:18.453397 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453386 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" containerName="manager" Jul 18 11:57:18.453437 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453399 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" containerName="manager" Jul 18 11:57:18.453437 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453421 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" containerName="manager" Jul 18 11:57:18.453437 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453426 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" containerName="manager" Jul 18 11:57:18.453733 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453477 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0c4284-7ca0-4e34-9d75-759a35d52c2a" containerName="manager" Jul 18 11:57:18.453733 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.453487 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6fab89b-c8c0-4273-b2dd-2fe76add906b" containerName="manager" Jul 18 11:57:18.455546 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.455522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.457878 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.457863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.460341 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.460323 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 11:57:18.460429 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.460344 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 18 11:57:18.461159 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.461139 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrrx5\"" Jul 18 11:57:18.461252 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.461148 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 11:57:18.465200 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.465184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:18.520301 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.520271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tslz9\" (UniqueName: \"kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.520515 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.520487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.550703 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.550674 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:18.554343 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.554316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.606913 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.606881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:18.621885 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.621863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tslz9\" (UniqueName: \"kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.621961 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.621899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.622538 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.622521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.629930 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.629909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tslz9\" (UniqueName: \"kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9\") pod \"limitador-limitador-64c94fd64f-ns74s\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.768615 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.768594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:18.838783 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.838753 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:57:18.842451 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.842425 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:18.849055 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.848981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:57:18.880958 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.880933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:57:18.906920 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.906900 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:18.908901 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:57:18.908865 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0c627b_d769_4fc6_9686_514ded12f58d.slice/crio-19d39233d62a76a0227f409a82fe4c23632d86d801684d5ee5863cad7b5eb187 WatchSource:0}: Error finding container 19d39233d62a76a0227f409a82fe4c23632d86d801684d5ee5863cad7b5eb187: Status 404 returned error can't find the container with id 19d39233d62a76a0227f409a82fe4c23632d86d801684d5ee5863cad7b5eb187 Jul 18 11:57:18.920092 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.920068 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" event={"ID":"8a0c627b-d769-4fc6-9686-514ded12f58d","Type":"ContainerStarted","Data":"19d39233d62a76a0227f409a82fe4c23632d86d801684d5ee5863cad7b5eb187"} Jul 18 11:57:18.924518 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.924498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704f4b8a-95a0-4168-9044-ac0c8450c992-config-file\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:18.924599 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:18.924528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vtz\" (UniqueName: \"kubernetes.io/projected/704f4b8a-95a0-4168-9044-ac0c8450c992-kube-api-access-r9vtz\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.025486 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.025404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704f4b8a-95a0-4168-9044-ac0c8450c992-config-file\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.025633 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.025605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vtz\" (UniqueName: \"kubernetes.io/projected/704f4b8a-95a0-4168-9044-ac0c8450c992-kube-api-access-r9vtz\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.026027 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.026006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/704f4b8a-95a0-4168-9044-ac0c8450c992-config-file\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.033463 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.033442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vtz\" (UniqueName: \"kubernetes.io/projected/704f4b8a-95a0-4168-9044-ac0c8450c992-kube-api-access-r9vtz\") pod \"limitador-limitador-54d6d988d5-pmtj9\" (UID: \"704f4b8a-95a0-4168-9044-ac0c8450c992\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.155371 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.155334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:19.298686 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.298663 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:57:19.300654 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:57:19.300627 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704f4b8a_95a0_4168_9044_ac0c8450c992.slice/crio-3371aeea1029951454997c02eb65ea9905f2dbf30e8d21c7db20f13023c5612c WatchSource:0}: Error finding container 3371aeea1029951454997c02eb65ea9905f2dbf30e8d21c7db20f13023c5612c: Status 404 returned error can't find the container with id 3371aeea1029951454997c02eb65ea9905f2dbf30e8d21c7db20f13023c5612c Jul 18 11:57:19.359931 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.359909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:19.421046 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.421023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:19.929219 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:19.929176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" event={"ID":"704f4b8a-95a0-4168-9044-ac0c8450c992","Type":"ContainerStarted","Data":"3371aeea1029951454997c02eb65ea9905f2dbf30e8d21c7db20f13023c5612c"} Jul 18 11:57:22.944796 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.944759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" event={"ID":"8a0c627b-d769-4fc6-9686-514ded12f58d","Type":"ContainerStarted","Data":"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97"} Jul 18 11:57:22.945449 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.944876 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:22.946968 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.946946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" event={"ID":"704f4b8a-95a0-4168-9044-ac0c8450c992","Type":"ContainerStarted","Data":"83fc1584fceefadb86172e3b4fc25f2551b7a90a3171f02f151898f5d0144f91"} Jul 18 11:57:22.947071 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.946989 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:22.960347 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.960304 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" podStartSLOduration=1.9336575630000001 podStartE2EDuration="4.960290796s" totalImagesPullingTime="3.026633233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:22.959505194 +0000 UTC m=+495.450277313" watchObservedRunningTime="2026-07-18 11:57:22.960290796 +0000 UTC m=+495.451062912" Jul 18 11:57:22.976268 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:22.976213 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" podStartSLOduration=2.334444391 podStartE2EDuration="4.97618914s" totalImagesPullingTime="2.641744749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:22.974769873 +0000 UTC m=+495.465542013" watchObservedRunningTime="2026-07-18 11:57:22.97618914 +0000 UTC m=+495.466961258" Jul 18 11:57:33.954016 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:33.953988 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:33.954404 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:33.954041 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-pmtj9" Jul 18 11:57:34.016513 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.016482 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:34.016677 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.016654 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" podUID="8a0c627b-d769-4fc6-9686-514ded12f58d" containerName="limitador" containerID="cri-o://9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97" gracePeriod=30 Jul 18 11:57:34.554648 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.554624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:34.661233 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.661166 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file\") pod \"8a0c627b-d769-4fc6-9686-514ded12f58d\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " Jul 18 11:57:34.661233 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.661222 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tslz9\" (UniqueName: \"kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9\") pod \"8a0c627b-d769-4fc6-9686-514ded12f58d\" (UID: \"8a0c627b-d769-4fc6-9686-514ded12f58d\") " Jul 18 11:57:34.661514 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.661487 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file" (OuterVolumeSpecName: "config-file") pod "8a0c627b-d769-4fc6-9686-514ded12f58d" (UID: "8a0c627b-d769-4fc6-9686-514ded12f58d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 11:57:34.663336 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.663313 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9" (OuterVolumeSpecName: "kube-api-access-tslz9") pod "8a0c627b-d769-4fc6-9686-514ded12f58d" (UID: "8a0c627b-d769-4fc6-9686-514ded12f58d"). InnerVolumeSpecName "kube-api-access-tslz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:57:34.762049 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.762020 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a0c627b-d769-4fc6-9686-514ded12f58d-config-file\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:57:34.762049 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.762046 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tslz9\" (UniqueName: \"kubernetes.io/projected/8a0c627b-d769-4fc6-9686-514ded12f58d-kube-api-access-tslz9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:57:34.995683 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.995644 2573 generic.go:356] "Generic (PLEG): container finished" podID="8a0c627b-d769-4fc6-9686-514ded12f58d" containerID="9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97" exitCode=0 Jul 18 11:57:34.996129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.995700 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" Jul 18 11:57:34.996129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.995732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" event={"ID":"8a0c627b-d769-4fc6-9686-514ded12f58d","Type":"ContainerDied","Data":"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97"} Jul 18 11:57:34.996129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.995773 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns74s" event={"ID":"8a0c627b-d769-4fc6-9686-514ded12f58d","Type":"ContainerDied","Data":"19d39233d62a76a0227f409a82fe4c23632d86d801684d5ee5863cad7b5eb187"} Jul 18 11:57:34.996129 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:34.995794 2573 scope.go:117] "RemoveContainer" containerID="9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97" Jul 18 11:57:35.004937 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:35.004915 2573 scope.go:117] "RemoveContainer" containerID="9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97" Jul 18 11:57:35.005167 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:57:35.005148 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97\": container with ID starting with 9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97 not found: ID does not exist" containerID="9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97" Jul 18 11:57:35.005226 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:35.005175 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97"} err="failed to get container status \"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97\": rpc error: code = NotFound desc = could not find container \"9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97\": container with ID starting with 9b8a08c998b3b7a576ce8e0dfd1c495086f4e8f521c134a719ae08c419cb5b97 not found: ID does not exist" Jul 18 11:57:35.016772 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:35.016751 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:35.022622 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:35.022595 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns74s"] Jul 18 11:57:35.954282 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:35.954250 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0c627b-d769-4fc6-9686-514ded12f58d" path="/var/lib/kubelet/pods/8a0c627b-d769-4fc6-9686-514ded12f58d/volumes" Jul 18 11:57:36.399356 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.399312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:36.496694 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.496663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:36.798338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.798308 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc"] Jul 18 11:57:36.798649 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.798638 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0c627b-d769-4fc6-9686-514ded12f58d" containerName="limitador" Jul 18 11:57:36.798694 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.798651 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0c627b-d769-4fc6-9686-514ded12f58d" containerName="limitador" Jul 18 11:57:36.798756 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.798725 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0c627b-d769-4fc6-9686-514ded12f58d" containerName="limitador" Jul 18 11:57:36.802956 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.802937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.805650 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.805622 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 11:57:36.805650 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.805633 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 18 11:57:36.806492 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.806473 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 11:57:36.806492 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.806488 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mvdqc\"" Jul 18 11:57:36.809592 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.809573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc"] Jul 18 11:57:36.832310 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.832289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:36.879248 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.879226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz72b\" (UniqueName: \"kubernetes.io/projected/d37511ee-734a-4842-ad9f-74fdaf4153a5-kube-api-access-cz72b\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.879337 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.879254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d37511ee-734a-4842-ad9f-74fdaf4153a5-data\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.980077 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.980045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz72b\" (UniqueName: \"kubernetes.io/projected/d37511ee-734a-4842-ad9f-74fdaf4153a5-kube-api-access-cz72b\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.980077 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.980076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d37511ee-734a-4842-ad9f-74fdaf4153a5-data\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.980385 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.980369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d37511ee-734a-4842-ad9f-74fdaf4153a5-data\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:36.988463 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:36.988436 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz72b\" (UniqueName: \"kubernetes.io/projected/d37511ee-734a-4842-ad9f-74fdaf4153a5-kube-api-access-cz72b\") pod \"postgres-69fb4594cf-5g5rc\" (UID: \"d37511ee-734a-4842-ad9f-74fdaf4153a5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:37.113667 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:37.113604 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:37.242951 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:37.242927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc"] Jul 18 11:57:37.245500 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:57:37.245474 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37511ee_734a_4842_ad9f_74fdaf4153a5.slice/crio-3579b7694dbfca601f9a5333aac12a5063b3f76d78a3f121d999a78e0a251034 WatchSource:0}: Error finding container 3579b7694dbfca601f9a5333aac12a5063b3f76d78a3f121d999a78e0a251034: Status 404 returned error can't find the container with id 3579b7694dbfca601f9a5333aac12a5063b3f76d78a3f121d999a78e0a251034 Jul 18 11:57:38.012536 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:38.012499 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" event={"ID":"d37511ee-734a-4842-ad9f-74fdaf4153a5","Type":"ContainerStarted","Data":"3579b7694dbfca601f9a5333aac12a5063b3f76d78a3f121d999a78e0a251034"} Jul 18 11:57:42.036964 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:42.036921 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" event={"ID":"d37511ee-734a-4842-ad9f-74fdaf4153a5","Type":"ContainerStarted","Data":"b669e0de12d71f4d9d4d965105df8ca75cf40930040f19d60d23a66d8848bf6c"} Jul 18 11:57:42.037320 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:42.037082 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:42.052555 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:42.052509 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" podStartSLOduration=1.351365678 podStartE2EDuration="6.052493202s" totalImagesPullingTime="4.701127524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:42.051258466 +0000 UTC m=+514.542030608" watchObservedRunningTime="2026-07-18 11:57:42.052493202 +0000 UTC m=+514.543265319" Jul 18 11:57:48.070128 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:48.070101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5g5rc" Jul 18 11:57:48.369679 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:48.369592 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:48.643630 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:48.643537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:54.594030 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.594002 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:54.607515 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.607489 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:57:54.631232 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.631210 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:57:54.631458 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.631296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.634048 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.634015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 11:57:54.634181 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.634051 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 11:57:54.634181 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.634171 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fwm9\"" Jul 18 11:57:54.694668 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.694639 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:54.723598 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.723568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqlp\" (UniqueName: \"kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.723744 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.723620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.763668 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.763645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:54.770910 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.770888 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 11:57:54.777216 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.775543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.786015 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.785989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 11:57:54.824866 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.824841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.824980 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.824880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqlp\" (UniqueName: \"kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.824980 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.824916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqwv\" (UniqueName: \"kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.824980 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.824955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.827347 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.827320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.832505 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.832482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqlp\" (UniqueName: \"kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp\") pod \"maas-controller-5b48dd7869-tvv8b\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:54.848401 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.848336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:54.925309 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.925276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.925425 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.925330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqwv\" (UniqueName: \"kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.927769 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.927747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.934000 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.933982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqwv\" (UniqueName: \"kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv\") pod \"maas-controller-84ff5bd5f9-jbs44\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:54.942779 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:54.942763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:55.069094 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:55.069037 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:57:55.071078 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:57:55.071055 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8a7c27_66be_44c6_a06f_737eb1acfe0c.slice/crio-e75b433487893027ab6e48dd8c615705a17d292af5f8a2cb140364badfde4ac5 WatchSource:0}: Error finding container e75b433487893027ab6e48dd8c615705a17d292af5f8a2cb140364badfde4ac5: Status 404 returned error can't find the container with id e75b433487893027ab6e48dd8c615705a17d292af5f8a2cb140364badfde4ac5 Jul 18 11:57:55.087338 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:55.087320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:55.090900 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:55.090874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" event={"ID":"9c8a7c27-66be-44c6-a06f-737eb1acfe0c","Type":"ContainerStarted","Data":"e75b433487893027ab6e48dd8c615705a17d292af5f8a2cb140364badfde4ac5"} Jul 18 11:57:55.218029 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:55.217997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 11:57:55.221106 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:57:55.221078 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efccfac_fe6b_4239_9093_8586fc3fc89a.slice/crio-cf7dc12c6887ae327d16e56cff04d7d82066eee4416503ab3cdd5a4100c7ffe8 WatchSource:0}: Error finding container cf7dc12c6887ae327d16e56cff04d7d82066eee4416503ab3cdd5a4100c7ffe8: Status 404 returned error can't find the container with id cf7dc12c6887ae327d16e56cff04d7d82066eee4416503ab3cdd5a4100c7ffe8 Jul 18 11:57:56.101166 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:56.101084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" event={"ID":"0efccfac-fe6b-4239-9093-8586fc3fc89a","Type":"ContainerStarted","Data":"cf7dc12c6887ae327d16e56cff04d7d82066eee4416503ab3cdd5a4100c7ffe8"} Jul 18 11:57:57.763233 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:57.763199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:57.899664 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:57.899581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:57:58.112358 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.112320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" event={"ID":"0efccfac-fe6b-4239-9093-8586fc3fc89a","Type":"ContainerStarted","Data":"10141b8aa1bf1135950729ad03ba723e7104c41f97475b511934be34fdcc1df1"} Jul 18 11:57:58.112562 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.112469 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:57:58.113850 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.113828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" event={"ID":"9c8a7c27-66be-44c6-a06f-737eb1acfe0c","Type":"ContainerStarted","Data":"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89"} Jul 18 11:57:58.113973 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.113954 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:57:58.128322 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.128279 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" podStartSLOduration=1.777476246 podStartE2EDuration="4.128263205s" totalImagesPullingTime="2.350786959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:58.127092806 +0000 UTC m=+530.617864923" watchObservedRunningTime="2026-07-18 11:57:58.128263205 +0000 UTC m=+530.619035322" Jul 18 11:57:58.142695 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:57:58.142657 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" podStartSLOduration=1.651111126 podStartE2EDuration="4.142643265s" totalImagesPullingTime="2.491532139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:58.141967267 +0000 UTC m=+530.632739510" watchObservedRunningTime="2026-07-18 11:57:58.142643265 +0000 UTC m=+530.633415381" Jul 18 11:58:09.121141 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.121113 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:58:09.121551 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.121186 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 11:58:09.175407 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.175377 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:58:09.175593 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.175554 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" podUID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" containerName="manager" containerID="cri-o://e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89" gracePeriod=30 Jul 18 11:58:09.418260 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.418236 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:58:09.544295 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.544263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert\") pod \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " Jul 18 11:58:09.544458 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.544305 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glqlp\" (UniqueName: \"kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp\") pod \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\" (UID: \"9c8a7c27-66be-44c6-a06f-737eb1acfe0c\") " Jul 18 11:58:09.546674 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.546647 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp" (OuterVolumeSpecName: "kube-api-access-glqlp") pod "9c8a7c27-66be-44c6-a06f-737eb1acfe0c" (UID: "9c8a7c27-66be-44c6-a06f-737eb1acfe0c"). InnerVolumeSpecName "kube-api-access-glqlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:58:09.546784 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.546696 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9c8a7c27-66be-44c6-a06f-737eb1acfe0c" (UID: "9c8a7c27-66be-44c6-a06f-737eb1acfe0c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 11:58:09.645283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.645246 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-webhook-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:58:09.645283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:09.645276 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqlp\" (UniqueName: \"kubernetes.io/projected/9c8a7c27-66be-44c6-a06f-737eb1acfe0c-kube-api-access-glqlp\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 11:58:10.170886 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.170850 2573 generic.go:356] "Generic (PLEG): container finished" podID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" containerID="e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89" exitCode=0 Jul 18 11:58:10.171258 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.170910 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" Jul 18 11:58:10.171258 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.170934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" event={"ID":"9c8a7c27-66be-44c6-a06f-737eb1acfe0c","Type":"ContainerDied","Data":"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89"} Jul 18 11:58:10.171258 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.171014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b48dd7869-tvv8b" event={"ID":"9c8a7c27-66be-44c6-a06f-737eb1acfe0c","Type":"ContainerDied","Data":"e75b433487893027ab6e48dd8c615705a17d292af5f8a2cb140364badfde4ac5"} Jul 18 11:58:10.171258 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.171038 2573 scope.go:117] "RemoveContainer" containerID="e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89" Jul 18 11:58:10.179295 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.179118 2573 scope.go:117] "RemoveContainer" containerID="e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89" Jul 18 11:58:10.179399 ip-10-0-137-172 kubenswrapper[2573]: E0718 11:58:10.179382 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89\": container with ID starting with e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89 not found: ID does not exist" containerID="e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89" Jul 18 11:58:10.179439 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.179407 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89"} err="failed to get container status \"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89\": rpc error: code = NotFound desc = could not find container \"e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89\": container with ID starting with e396b03ad87fdf41c97572fb4f120332780156e652891eb813a6d653edee1f89 not found: ID does not exist" Jul 18 11:58:10.187356 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.187336 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:58:10.189727 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:10.189686 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b48dd7869-tvv8b"] Jul 18 11:58:11.954637 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:11.954603 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" path="/var/lib/kubelet/pods/9c8a7c27-66be-44c6-a06f-737eb1acfe0c/volumes" Jul 18 11:58:30.885605 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:30.885566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:30.982348 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:30.982314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:31.503423 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.503390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:31.612547 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.612512 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r"] Jul 18 11:58:31.612930 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.612913 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" containerName="manager" Jul 18 11:58:31.613031 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.612933 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" containerName="manager" Jul 18 11:58:31.613031 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.613017 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8a7c27-66be-44c6-a06f-737eb1acfe0c" containerName="manager" Jul 18 11:58:31.620461 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.620441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.623811 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.623788 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9584k\"" Jul 18 11:58:31.623913 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.623814 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 18 11:58:31.627542 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.627284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r"] Jul 18 11:58:31.633375 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.633347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:31.726810 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.726776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqn8\" (UniqueName: \"kubernetes.io/projected/8a59803c-b759-4bdd-92f0-620437320966-kube-api-access-mxqn8\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.726955 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.726817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a59803c-b759-4bdd-92f0-620437320966-openshift-service-ca\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.726955 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.726883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a59803c-b759-4bdd-92f0-620437320966-maas-api-tls\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.828079 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.827996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a59803c-b759-4bdd-92f0-620437320966-openshift-service-ca\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.828079 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.828051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a59803c-b759-4bdd-92f0-620437320966-maas-api-tls\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.828298 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.828127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqn8\" (UniqueName: \"kubernetes.io/projected/8a59803c-b759-4bdd-92f0-620437320966-kube-api-access-mxqn8\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.828620 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.828592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a59803c-b759-4bdd-92f0-620437320966-openshift-service-ca\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.830635 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.830616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a59803c-b759-4bdd-92f0-620437320966-maas-api-tls\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.836161 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.836133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqn8\" (UniqueName: \"kubernetes.io/projected/8a59803c-b759-4bdd-92f0-620437320966-kube-api-access-mxqn8\") pod \"maas-api-86bc78d86c-mdn8r\" (UID: \"8a59803c-b759-4bdd-92f0-620437320966\") " pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.936763 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.936732 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:31.963374 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:31.963346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:32.059843 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.059810 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pttt5"] Jul 18 11:58:32.063530 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.063503 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:32.063967 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.063953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.066262 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.066246 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 18 11:58:32.066373 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.066355 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9qbwk\"" Jul 18 11:58:32.080298 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.080222 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pttt5"] Jul 18 11:58:32.082343 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.082324 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r"] Jul 18 11:58:32.084439 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:58:32.084414 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a59803c_b759_4bdd_92f0_620437320966.slice/crio-0064417c2fdaf0a369f7b449678fbf633f079306465b51e59f8286e64a9492a1 WatchSource:0}: Error finding container 0064417c2fdaf0a369f7b449678fbf633f079306465b51e59f8286e64a9492a1: Status 404 returned error can't find the container with id 0064417c2fdaf0a369f7b449678fbf633f079306465b51e59f8286e64a9492a1 Jul 18 11:58:32.167019 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.166995 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:58:32.231615 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.231582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbd4\" (UniqueName: \"kubernetes.io/projected/8886b023-80b5-46ad-8d30-edbf231cd91b-kube-api-access-tnbd4\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.231775 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.231620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8886b023-80b5-46ad-8d30-edbf231cd91b-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.264273 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.264243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" event={"ID":"8a59803c-b759-4bdd-92f0-620437320966","Type":"ContainerStarted","Data":"0064417c2fdaf0a369f7b449678fbf633f079306465b51e59f8286e64a9492a1"} Jul 18 11:58:32.332754 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.332659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbd4\" (UniqueName: \"kubernetes.io/projected/8886b023-80b5-46ad-8d30-edbf231cd91b-kube-api-access-tnbd4\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.332754 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.332691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8886b023-80b5-46ad-8d30-edbf231cd91b-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.333604 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.333576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8886b023-80b5-46ad-8d30-edbf231cd91b-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.346641 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.346621 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbd4\" (UniqueName: \"kubernetes.io/projected/8886b023-80b5-46ad-8d30-edbf231cd91b-kube-api-access-tnbd4\") pod \"payload-pre-processing-7677b774bb-pttt5\" (UID: \"8886b023-80b5-46ad-8d30-edbf231cd91b\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.374571 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.374545 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:32.503037 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:32.502985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-pttt5"] Jul 18 11:58:32.505473 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:58:32.505448 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8886b023_80b5_46ad_8d30_edbf231cd91b.slice/crio-1ed30c050bc73b5305f4b7d7d783465bc5bb330b0cbc8e7843d72c6f730542e6 WatchSource:0}: Error finding container 1ed30c050bc73b5305f4b7d7d783465bc5bb330b0cbc8e7843d72c6f730542e6: Status 404 returned error can't find the container with id 1ed30c050bc73b5305f4b7d7d783465bc5bb330b0cbc8e7843d72c6f730542e6 Jul 18 11:58:33.276512 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:33.276475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" event={"ID":"8886b023-80b5-46ad-8d30-edbf231cd91b","Type":"ContainerStarted","Data":"1ed30c050bc73b5305f4b7d7d783465bc5bb330b0cbc8e7843d72c6f730542e6"} Jul 18 11:58:34.285674 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:34.285640 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" event={"ID":"8a59803c-b759-4bdd-92f0-620437320966","Type":"ContainerStarted","Data":"18453a3ea33f99ffda818fa1e476924811adf04ee5befee9c2d45e42534d3073"} Jul 18 11:58:34.304029 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:34.303971 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" podStartSLOduration=1.790640375 podStartE2EDuration="3.303956866s" totalImagesPullingTime="1.513316491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:58:34.302273084 +0000 UTC m=+566.793045203" watchObservedRunningTime="2026-07-18 11:58:34.303956866 +0000 UTC m=+566.794728983" Jul 18 11:58:35.291954 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:35.291919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" event={"ID":"8886b023-80b5-46ad-8d30-edbf231cd91b","Type":"ContainerStarted","Data":"911ff42bfdd5cfb6f007587fc213315980ee75634ad10031bd2079f2a1f81e2b"} Jul 18 11:58:35.292326 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:35.292034 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:58:35.292326 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:35.292085 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:35.306959 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:35.306848 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" podStartSLOduration=1.047254841 podStartE2EDuration="3.30683252s" totalImagesPullingTime="2.259577679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:58:35.306204428 +0000 UTC m=+567.796976544" watchObservedRunningTime="2026-07-18 11:58:35.30683252 +0000 UTC m=+567.797604640" Jul 18 11:58:41.301997 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:41.301968 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86bc78d86c-mdn8r" Jul 18 11:58:42.276419 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:42.276371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:42.376266 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:42.376228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:58:46.298852 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:58:46.298820 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-pttt5" Jul 18 11:59:00.893368 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:00.893331 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:00.950025 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:00.949992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:02.047392 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:02.047362 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:02.094671 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:02.094637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:05.159544 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.159512 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs"] Jul 18 11:59:05.165538 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.165516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.169045 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.169022 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 11:59:05.169045 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.169033 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 18 11:59:05.169189 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.169054 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 11:59:05.169189 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.169038 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k9r9z\"" Jul 18 11:59:05.173671 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.173211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs"] Jul 18 11:59:05.180590 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.180564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:05.315691 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.315691 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.315966 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhk4f\" (UniqueName: \"kubernetes.io/projected/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-kube-api-access-vhk4f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.315966 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.315966 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.315966 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.315947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417283 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhk4f\" (UniqueName: \"kubernetes.io/projected/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-kube-api-access-vhk4f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417545 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417545 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417545 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.417852 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417824 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.418151 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417865 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.418151 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.417916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.419627 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.419607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.420156 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.420134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.425384 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.425363 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhk4f\" (UniqueName: \"kubernetes.io/projected/cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144-kube-api-access-vhk4f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs\" (UID: \"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.478418 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.478393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:05.567943 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.567910 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:59:05.612742 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:05.612662 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs"] Jul 18 11:59:05.615169 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:59:05.615143 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7bb9f5_5abe_4a8d_a3dc_9c4e28c51144.slice/crio-1b4b252faa288636676f0ec7b0a8b912ec8418598767a0414f44c21e322603c5 WatchSource:0}: Error finding container 1b4b252faa288636676f0ec7b0a8b912ec8418598767a0414f44c21e322603c5: Status 404 returned error can't find the container with id 1b4b252faa288636676f0ec7b0a8b912ec8418598767a0414f44c21e322603c5 Jul 18 11:59:06.427871 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:06.427829 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" event={"ID":"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144","Type":"ContainerStarted","Data":"1b4b252faa288636676f0ec7b0a8b912ec8418598767a0414f44c21e322603c5"} Jul 18 11:59:07.434270 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:07.434236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" event={"ID":"cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144","Type":"ContainerStarted","Data":"b17977f0e799a9d1cdd1e0931d4fd29b0d52b17b97959cf2b4df07cb7880e3c0"} Jul 18 11:59:07.451588 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:07.451530 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" podStartSLOduration=1.44409142 podStartE2EDuration="2.451510731s" totalImagesPullingTime="1.007419311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:59:07.450342323 +0000 UTC m=+599.941114442" watchObservedRunningTime="2026-07-18 11:59:07.451510731 +0000 UTC m=+599.942282826" Jul 18 11:59:07.865321 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:07.865300 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:59:07.865506 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:07.865487 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 11:59:09.964468 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:09.964430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:10.679033 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:10.678995 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:59:15.478809 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:15.478778 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:15.479197 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:15.478852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:15.491227 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:15.491206 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:16.490154 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:16.490125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs" Jul 18 11:59:18.670998 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:18.670962 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:19.676847 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:19.676815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:20.970489 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:20.970454 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:59:33.550029 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.549985 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7"] Jul 18 11:59:33.553425 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.553405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.555898 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.555880 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 18 11:59:33.564567 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.564546 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7"] Jul 18 11:59:33.659589 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.659807 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.659807 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.659807 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88tl\" (UniqueName: \"kubernetes.io/projected/b4995746-508e-4b6c-b0a3-864fc4ccf55e-kube-api-access-h88tl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.659807 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.659807 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.659762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.661476 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.661449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:33.760962 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.760923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761119 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.760970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88tl\" (UniqueName: \"kubernetes.io/projected/b4995746-508e-4b6c-b0a3-864fc4ccf55e-kube-api-access-h88tl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761119 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761119 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761119 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761320 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761320 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761422 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.761494 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.761475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.763430 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.763409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4995746-508e-4b6c-b0a3-864fc4ccf55e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.763574 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.763557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4995746-508e-4b6c-b0a3-864fc4ccf55e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.768549 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.768528 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88tl\" (UniqueName: \"kubernetes.io/projected/b4995746-508e-4b6c-b0a3-864fc4ccf55e-kube-api-access-h88tl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7\" (UID: \"b4995746-508e-4b6c-b0a3-864fc4ccf55e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.862671 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.862603 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:33.994544 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.994517 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7"] Jul 18 11:59:33.997300 ip-10-0-137-172 kubenswrapper[2573]: W0718 11:59:33.997258 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4995746_508e_4b6c_b0a3_864fc4ccf55e.slice/crio-4a51ed11a89701b88cb984c09b280c30ca0c4d9415cde7740017323f0bf7252f WatchSource:0}: Error finding container 4a51ed11a89701b88cb984c09b280c30ca0c4d9415cde7740017323f0bf7252f: Status 404 returned error can't find the container with id 4a51ed11a89701b88cb984c09b280c30ca0c4d9415cde7740017323f0bf7252f Jul 18 11:59:33.998602 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:33.998586 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:59:34.558942 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:34.558909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" event={"ID":"b4995746-508e-4b6c-b0a3-864fc4ccf55e","Type":"ContainerStarted","Data":"f34b20dffc396532fd6b89015ddb66808a9d0481e125ddf478704aef871b28bb"} Jul 18 11:59:34.559399 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:34.558970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" event={"ID":"b4995746-508e-4b6c-b0a3-864fc4ccf55e","Type":"ContainerStarted","Data":"4a51ed11a89701b88cb984c09b280c30ca0c4d9415cde7740017323f0bf7252f"} Jul 18 11:59:34.577055 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:34.577010 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" podStartSLOduration=1.420023689 podStartE2EDuration="1.576996714s" totalImagesPullingTime="156.973025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:59:34.575406143 +0000 UTC m=+627.066178260" watchObservedRunningTime="2026-07-18 11:59:34.576996714 +0000 UTC m=+627.067768831" Jul 18 11:59:34.977317 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:34.977239 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:59:36.948825 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:36.948786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:41.969450 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:41.969420 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 11:59:43.862810 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:43.862778 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:43.863192 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:43.862841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:43.875265 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:43.875238 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:44.612374 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:44.612337 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7" Jul 18 11:59:45.781339 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:45.781305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 11:59:48.674433 ip-10-0-137-172 kubenswrapper[2573]: I0718 11:59:48.674400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:00:00.146102 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.146068 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns"] Jul 18 12:00:00.149320 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.149305 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" Jul 18 12:00:00.151883 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.151857 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:00:00.157837 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.157810 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns"] Jul 18 12:00:00.160662 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.160638 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" Jul 18 12:00:00.299860 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.299836 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns"] Jul 18 12:00:00.301997 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:00:00.301952 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae6bda8_e5be_4d99_a704_2ecf584f2935.slice/crio-a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567 WatchSource:0}: Error finding container a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567: Status 404 returned error can't find the container with id a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567 Jul 18 12:00:00.674317 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:00.674285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" event={"ID":"cae6bda8-e5be-4d99-a704-2ecf584f2935","Type":"ContainerStarted","Data":"a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567"} Jul 18 12:00:05.698519 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:05.698435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" event={"ID":"cae6bda8-e5be-4d99-a704-2ecf584f2935","Type":"ContainerStarted","Data":"78209f4877993e2aee4b1ac0709fba4dd685c60c1558f23d941db194844498a5"} Jul 18 12:00:05.715357 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:05.715302 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" podStartSLOduration=0.999498714 podStartE2EDuration="5.715286735s" totalImagesPullingTime="4.715788021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:00:05.713687728 +0000 UTC m=+658.204459845" watchObservedRunningTime="2026-07-18 12:00:05.715286735 +0000 UTC m=+658.206058851" Jul 18 12:00:06.704147 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:06.704115 2573 generic.go:356] "Generic (PLEG): container finished" podID="cae6bda8-e5be-4d99-a704-2ecf584f2935" containerID="78209f4877993e2aee4b1ac0709fba4dd685c60c1558f23d941db194844498a5" exitCode=0 Jul 18 12:00:06.704553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:06.704208 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" event={"ID":"cae6bda8-e5be-4d99-a704-2ecf584f2935","Type":"ContainerDied","Data":"78209f4877993e2aee4b1ac0709fba4dd685c60c1558f23d941db194844498a5"} Jul 18 12:00:07.849368 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:07.849341 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" Jul 18 12:00:08.719359 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:08.719325 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" Jul 18 12:00:08.719359 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:08.719344 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739600-k2cns" event={"ID":"cae6bda8-e5be-4d99-a704-2ecf584f2935","Type":"ContainerDied","Data":"a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567"} Jul 18 12:00:08.719563 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:00:08.719390 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8043cc1aa26f033761e2830c5c4ccd50e4d5c1ede8aab1bec8f2babc86fb567" Jul 18 12:01:17.752872 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:17.752769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:01:17.817984 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:17.817953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:01:18.807494 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:18.807454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:01:18.918783 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:18.918703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:01:45.974079 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:45.974049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:01:55.465205 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:01:55.465172 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:02:03.101550 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.101506 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 12:02:03.101978 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.101740 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" podUID="0efccfac-fe6b-4239-9093-8586fc3fc89a" containerName="manager" containerID="cri-o://10141b8aa1bf1135950729ad03ba723e7104c41f97475b511934be34fdcc1df1" gracePeriod=30 Jul 18 12:02:03.220777 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.220748 2573 generic.go:356] "Generic (PLEG): container finished" podID="0efccfac-fe6b-4239-9093-8586fc3fc89a" containerID="10141b8aa1bf1135950729ad03ba723e7104c41f97475b511934be34fdcc1df1" exitCode=0 Jul 18 12:02:03.220879 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.220821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" event={"ID":"0efccfac-fe6b-4239-9093-8586fc3fc89a","Type":"ContainerDied","Data":"10141b8aa1bf1135950729ad03ba723e7104c41f97475b511934be34fdcc1df1"} Jul 18 12:02:03.343932 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.343909 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 12:02:03.476584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.476497 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqwv\" (UniqueName: \"kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv\") pod \"0efccfac-fe6b-4239-9093-8586fc3fc89a\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " Jul 18 12:02:03.476584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.476566 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert\") pod \"0efccfac-fe6b-4239-9093-8586fc3fc89a\" (UID: \"0efccfac-fe6b-4239-9093-8586fc3fc89a\") " Jul 18 12:02:03.478661 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.478629 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv" (OuterVolumeSpecName: "kube-api-access-4nqwv") pod "0efccfac-fe6b-4239-9093-8586fc3fc89a" (UID: "0efccfac-fe6b-4239-9093-8586fc3fc89a"). InnerVolumeSpecName "kube-api-access-4nqwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:02:03.478661 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.478631 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0efccfac-fe6b-4239-9093-8586fc3fc89a" (UID: "0efccfac-fe6b-4239-9093-8586fc3fc89a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:02:03.577882 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.577847 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqwv\" (UniqueName: \"kubernetes.io/projected/0efccfac-fe6b-4239-9093-8586fc3fc89a-kube-api-access-4nqwv\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:02:03.577882 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:03.577875 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0efccfac-fe6b-4239-9093-8586fc3fc89a-webhook-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:02:04.226190 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:04.226102 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" Jul 18 12:02:04.226190 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:04.226132 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-jbs44" event={"ID":"0efccfac-fe6b-4239-9093-8586fc3fc89a","Type":"ContainerDied","Data":"cf7dc12c6887ae327d16e56cff04d7d82066eee4416503ab3cdd5a4100c7ffe8"} Jul 18 12:02:04.226663 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:04.226194 2573 scope.go:117] "RemoveContainer" containerID="10141b8aa1bf1135950729ad03ba723e7104c41f97475b511934be34fdcc1df1" Jul 18 12:02:04.243555 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:04.243531 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 12:02:04.248854 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:04.248830 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-jbs44"] Jul 18 12:02:05.869775 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:05.869732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:02:05.955381 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:02:05.955348 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efccfac-fe6b-4239-9093-8586fc3fc89a" path="/var/lib/kubelet/pods/0efccfac-fe6b-4239-9093-8586fc3fc89a/volumes" Jul 18 12:03:05.025275 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025234 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025571 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efccfac-fe6b-4239-9093-8586fc3fc89a" containerName="manager" Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025582 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efccfac-fe6b-4239-9093-8586fc3fc89a" containerName="manager" Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025605 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae6bda8-e5be-4d99-a704-2ecf584f2935" containerName="cleanup" Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025610 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae6bda8-e5be-4d99-a704-2ecf584f2935" containerName="cleanup" Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025661 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efccfac-fe6b-4239-9093-8586fc3fc89a" containerName="manager" Jul 18 12:03:05.025762 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.025671 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae6bda8-e5be-4d99-a704-2ecf584f2935" containerName="cleanup" Jul 18 12:03:05.028646 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.028625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.031929 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.031909 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fwm9\"" Jul 18 12:03:05.032045 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.031938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 12:03:05.032045 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.031950 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 12:03:05.037789 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.037765 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:05.117334 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.117301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7nd\" (UniqueName: \"kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.117507 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.117349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.218282 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.218248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7nd\" (UniqueName: \"kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.218441 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.218292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.220824 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.220802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.226358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.226335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7nd\" (UniqueName: \"kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd\") pod \"maas-controller-84ff5bd5f9-48h66\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.339502 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.339410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:05.469645 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.469583 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:05.472044 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:03:05.472017 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda08df63_c5d3_434b_9c84_2dd2ca3f540d.slice/crio-ed0598ffc12eb53b38f23c143fd7ef48a2261e3ba690406b5ca1eb19f888d2c0 WatchSource:0}: Error finding container ed0598ffc12eb53b38f23c143fd7ef48a2261e3ba690406b5ca1eb19f888d2c0: Status 404 returned error can't find the container with id ed0598ffc12eb53b38f23c143fd7ef48a2261e3ba690406b5ca1eb19f888d2c0 Jul 18 12:03:05.495990 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:05.495963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" event={"ID":"da08df63-c5d3-434b-9c84-2dd2ca3f540d","Type":"ContainerStarted","Data":"ed0598ffc12eb53b38f23c143fd7ef48a2261e3ba690406b5ca1eb19f888d2c0"} Jul 18 12:03:06.501973 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:06.501934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" event={"ID":"da08df63-c5d3-434b-9c84-2dd2ca3f540d","Type":"ContainerStarted","Data":"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09"} Jul 18 12:03:06.502392 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:06.502048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:06.518452 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:06.518361 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" podStartSLOduration=1.039096056 podStartE2EDuration="1.518345808s" totalImagesPullingTime="479.249752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:03:06.518189904 +0000 UTC m=+839.008962038" watchObservedRunningTime="2026-07-18 12:03:06.518345808 +0000 UTC m=+839.009117927" Jul 18 12:03:17.507688 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:17.507655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:22.049904 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.049870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:03:22.115662 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.115629 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:22.115894 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.115844 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" podUID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" containerName="manager" containerID="cri-o://9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09" gracePeriod=30 Jul 18 12:03:22.362969 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.362946 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:22.455018 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.454987 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg7nd\" (UniqueName: \"kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd\") pod \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " Jul 18 12:03:22.455174 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.455089 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert\") pod \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\" (UID: \"da08df63-c5d3-434b-9c84-2dd2ca3f540d\") " Jul 18 12:03:22.457145 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.457117 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "da08df63-c5d3-434b-9c84-2dd2ca3f540d" (UID: "da08df63-c5d3-434b-9c84-2dd2ca3f540d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:03:22.457252 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.457118 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd" (OuterVolumeSpecName: "kube-api-access-sg7nd") pod "da08df63-c5d3-434b-9c84-2dd2ca3f540d" (UID: "da08df63-c5d3-434b-9c84-2dd2ca3f540d"). InnerVolumeSpecName "kube-api-access-sg7nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:03:22.555962 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.555932 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da08df63-c5d3-434b-9c84-2dd2ca3f540d-webhook-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:03:22.555962 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.555956 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7nd\" (UniqueName: \"kubernetes.io/projected/da08df63-c5d3-434b-9c84-2dd2ca3f540d-kube-api-access-sg7nd\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:03:22.572599 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.572565 2573 generic.go:356] "Generic (PLEG): container finished" podID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" containerID="9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09" exitCode=0 Jul 18 12:03:22.572755 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.572601 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" event={"ID":"da08df63-c5d3-434b-9c84-2dd2ca3f540d","Type":"ContainerDied","Data":"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09"} Jul 18 12:03:22.572755 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.572608 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" Jul 18 12:03:22.572755 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.572656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-48h66" event={"ID":"da08df63-c5d3-434b-9c84-2dd2ca3f540d","Type":"ContainerDied","Data":"ed0598ffc12eb53b38f23c143fd7ef48a2261e3ba690406b5ca1eb19f888d2c0"} Jul 18 12:03:22.572755 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.572686 2573 scope.go:117] "RemoveContainer" containerID="9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09" Jul 18 12:03:22.581182 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.581166 2573 scope.go:117] "RemoveContainer" containerID="9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09" Jul 18 12:03:22.581418 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:03:22.581401 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09\": container with ID starting with 9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09 not found: ID does not exist" containerID="9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09" Jul 18 12:03:22.581458 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.581427 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09"} err="failed to get container status \"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09\": rpc error: code = NotFound desc = could not find container \"9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09\": container with ID starting with 9228d32e1ae93971555c91c8afa5d5a7fe556b97340b6de3ce49ec8f796a5a09 not found: ID does not exist" Jul 18 12:03:22.594965 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.594892 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:22.596597 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:22.596578 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-48h66"] Jul 18 12:03:23.243257 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.243226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:03:23.351195 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.351156 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-vqxz2"] Jul 18 12:03:23.351570 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.351553 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" containerName="manager" Jul 18 12:03:23.351665 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.351573 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" containerName="manager" Jul 18 12:03:23.351749 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.351678 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" containerName="manager" Jul 18 12:03:23.355798 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.355778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.358040 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.358020 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 12:03:23.359174 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.359157 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 12:03:23.359278 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.359157 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fwm9\"" Jul 18 12:03:23.361128 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.361100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07315cc8-1e86-4caa-88fc-d498a3f7f00c-webhook-cert\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.361234 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.361145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrckh\" (UniqueName: \"kubernetes.io/projected/07315cc8-1e86-4caa-88fc-d498a3f7f00c-kube-api-access-qrckh\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.364434 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.364411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-vqxz2"] Jul 18 12:03:23.462415 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.462382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07315cc8-1e86-4caa-88fc-d498a3f7f00c-webhook-cert\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.462415 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.462421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrckh\" (UniqueName: \"kubernetes.io/projected/07315cc8-1e86-4caa-88fc-d498a3f7f00c-kube-api-access-qrckh\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.464992 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.464971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07315cc8-1e86-4caa-88fc-d498a3f7f00c-webhook-cert\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.470411 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.470380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrckh\" (UniqueName: \"kubernetes.io/projected/07315cc8-1e86-4caa-88fc-d498a3f7f00c-kube-api-access-qrckh\") pod \"maas-controller-84ff5bd5f9-vqxz2\" (UID: \"07315cc8-1e86-4caa-88fc-d498a3f7f00c\") " pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.666834 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.666801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:23.795747 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.795723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84ff5bd5f9-vqxz2"] Jul 18 12:03:23.797456 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:03:23.797430 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07315cc8_1e86_4caa_88fc_d498a3f7f00c.slice/crio-7090b1d5f92d04166c8ef5a2d4e24abd5ca7c9c17444a43f5e82942def5aaeb2 WatchSource:0}: Error finding container 7090b1d5f92d04166c8ef5a2d4e24abd5ca7c9c17444a43f5e82942def5aaeb2: Status 404 returned error can't find the container with id 7090b1d5f92d04166c8ef5a2d4e24abd5ca7c9c17444a43f5e82942def5aaeb2 Jul 18 12:03:23.955659 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:23.955579 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da08df63-c5d3-434b-9c84-2dd2ca3f540d" path="/var/lib/kubelet/pods/da08df63-c5d3-434b-9c84-2dd2ca3f540d/volumes" Jul 18 12:03:24.583252 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:24.583216 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" event={"ID":"07315cc8-1e86-4caa-88fc-d498a3f7f00c","Type":"ContainerStarted","Data":"213e307cfc37bf35cb1d85b278d71e45b9cc7cfa7313e3fd239f8f128c6b5e80"} Jul 18 12:03:24.583766 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:24.583264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" event={"ID":"07315cc8-1e86-4caa-88fc-d498a3f7f00c","Type":"ContainerStarted","Data":"7090b1d5f92d04166c8ef5a2d4e24abd5ca7c9c17444a43f5e82942def5aaeb2"} Jul 18 12:03:24.583766 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:24.583351 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:24.615372 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:24.615291 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" podStartSLOduration=1.200395722 podStartE2EDuration="1.615277098s" totalImagesPullingTime="414.881376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:03:24.612761336 +0000 UTC m=+857.103533454" watchObservedRunningTime="2026-07-18 12:03:24.615277098 +0000 UTC m=+857.106049214" Jul 18 12:03:35.589911 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:35.589882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84ff5bd5f9-vqxz2" Jul 18 12:03:51.167768 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:51.167735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:03:58.567324 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:58.567236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:03:58.668660 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:58.668628 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:03:58.786977 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:03:58.786946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:00.263733 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:00.263684 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:04:04.553047 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:04.553013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:04.659763 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:04.659725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:04.754144 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:04.754113 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:04.898462 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:04.898374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:07.897537 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:07.897513 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:04:07.898777 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:07.898759 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:04:10.856797 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:10.856764 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:10.956872 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:10.956833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:15.540500 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:15.540464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:15.640316 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:15.640284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:04:27.559958 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:27.559923 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:04:37.261937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:37.261906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:04:45.769950 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:45.769914 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:04:56.974605 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:56.974569 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:04:59.703288 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:04:59.703250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:05:07.569165 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:07.569127 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:05:11.751760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:11.751699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:05:11.851874 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:11.851844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:05:18.573554 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:18.573523 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:05:29.463430 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:29.463392 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:05:39.867102 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:05:39.867066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:06:01.169943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:06:01.169909 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:06:13.069197 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:06:13.069167 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:06:21.979377 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:06:21.979330 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:06:36.566862 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:06:36.566827 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:06:47.275565 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:06:47.271059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:07:01.973363 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:07:01.973284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:07:13.374252 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:07:13.374215 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:07:22.874299 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:07:22.874266 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:07:35.069766 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:07:35.069734 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:09:07.924217 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:09:07.924190 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:09:07.932364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:09:07.932345 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:09:28.976169 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:09:28.976132 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:09:38.261842 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:09:38.261803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:09:51.173792 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:09:51.173762 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:02.466530 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:02.466444 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:12.695249 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:12.695204 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:24.174393 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:24.174355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:32.068627 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:32.068590 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:43.066538 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:43.066506 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:10:55.270530 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:10:55.270495 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:11:04.064830 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:11:04.064791 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:11:16.064145 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:11:16.064110 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:11:37.173565 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:11:37.173532 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:11:47.673263 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:11:47.673227 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:11:52.079657 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:11:52.079625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:12:02.370141 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:12:02.370057 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:12:05.070434 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:12:05.070394 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:12:35.547965 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:12:35.547928 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:12:46.122622 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:12:46.122588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:14:07.950695 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:14:07.950661 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:14:07.960890 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:14:07.960868 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:15:00.138238 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.138196 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq"] Jul 18 12:15:00.141253 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.141224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" Jul 18 12:15:00.143997 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.143978 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:15:00.150466 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.150445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" Jul 18 12:15:00.158685 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.158659 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq"] Jul 18 12:15:00.286892 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.286867 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq"] Jul 18 12:15:00.288944 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:15:00.288914 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb2aa02_b86a_432e_bf0e_5ed5978db9c2.slice/crio-6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c WatchSource:0}: Error finding container 6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c: Status 404 returned error can't find the container with id 6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c Jul 18 12:15:00.654398 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.654362 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" event={"ID":"3cb2aa02-b86a-432e-bf0e-5ed5978db9c2","Type":"ContainerStarted","Data":"0fec8fe882e7d7175887b62ec28b23d37f33e5ede6939a61b8980d65bf3afffe"} Jul 18 12:15:00.654581 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.654423 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" event={"ID":"3cb2aa02-b86a-432e-bf0e-5ed5978db9c2","Type":"ContainerStarted","Data":"6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c"} Jul 18 12:15:00.670985 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:00.670928 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" podStartSLOduration=0.67091221 podStartE2EDuration="670.91221ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:15:00.669258157 +0000 UTC m=+1553.160030275" watchObservedRunningTime="2026-07-18 12:15:00.67091221 +0000 UTC m=+1553.161684327" Jul 18 12:15:01.374550 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:01.374516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:15:01.660991 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:01.660953 2573 generic.go:356] "Generic (PLEG): container finished" podID="3cb2aa02-b86a-432e-bf0e-5ed5978db9c2" containerID="0fec8fe882e7d7175887b62ec28b23d37f33e5ede6939a61b8980d65bf3afffe" exitCode=0 Jul 18 12:15:01.661151 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:01.661016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" event={"ID":"3cb2aa02-b86a-432e-bf0e-5ed5978db9c2","Type":"ContainerDied","Data":"0fec8fe882e7d7175887b62ec28b23d37f33e5ede6939a61b8980d65bf3afffe"} Jul 18 12:15:02.069596 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:02.069568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:15:02.795608 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:02.795585 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" Jul 18 12:15:03.670653 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:03.670626 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" Jul 18 12:15:03.670653 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:03.670641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739615-f25sq" event={"ID":"3cb2aa02-b86a-432e-bf0e-5ed5978db9c2","Type":"ContainerDied","Data":"6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c"} Jul 18 12:15:03.670873 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:03.670689 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7ac5f6e98ffe76ca7b09eb3cb84511024e84b175bd6be6851ead336c34358c" Jul 18 12:15:04.878276 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:04.878239 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:15:18.568511 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:18.568470 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:15:25.468366 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:25.468325 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:15:40.677748 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:40.677146 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:15:49.673192 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:49.673155 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:15:58.976915 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:15:58.976835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:16:08.177141 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:16:08.177109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:16:19.277745 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:16:19.277695 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:16:29.073894 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:16:29.073861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:16:42.370804 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:16:42.370770 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:16:51.767031 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:16:51.766996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:01.476792 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:01.476760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:14.175584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:14.175540 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:25.464537 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:25.464475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:34.064857 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:34.064821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:46.168030 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:46.167992 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:17:57.369932 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:17:57.369896 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:18:06.471333 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:06.471293 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:18:13.868649 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:13.868611 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:18:19.172230 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:19.172193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:20.346776 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:20.346737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:25.116947 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:25.116908 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:25.217426 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:25.217393 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:25.344882 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:25.344845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:25.669997 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:25.669960 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:26.050189 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:26.050157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:27.505565 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.505526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:27.597506 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.597465 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:18:27.598048 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.598029 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb2aa02-b86a-432e-bf0e-5ed5978db9c2" containerName="cleanup" Jul 18 12:18:27.598117 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.598052 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb2aa02-b86a-432e-bf0e-5ed5978db9c2" containerName="cleanup" Jul 18 12:18:27.598193 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.598181 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb2aa02-b86a-432e-bf0e-5ed5978db9c2" containerName="cleanup" Jul 18 12:18:27.600111 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.600084 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:27.601744 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.601726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.604034 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.604013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-736936a8-serving-cert\"" Jul 18 12:18:27.610536 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.610517 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:18:27.670685 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.670654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:27.713952 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.713926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.714103 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.713965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.714103 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.714009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zx9k\" (UniqueName: \"kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.815362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.815280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.815362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.815353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zx9k\" (UniqueName: \"kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.815537 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.815408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.815537 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:18:27.815509 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-serving-cert: secret "maas-api-e2e-ait-736936a8-serving-cert" not found Jul 18 12:18:27.815606 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:18:27.815573 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls podName:d5d5661a-262d-4fe3-85ac-f9c5a45a481a nodeName:}" failed. No retries permitted until 2026-07-18 12:18:28.315553131 +0000 UTC m=+1760.806325225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls") pod "maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" (UID: "d5d5661a-262d-4fe3-85ac-f9c5a45a481a") : secret "maas-api-e2e-ait-736936a8-serving-cert" not found Jul 18 12:18:27.816062 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.816042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.829169 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.829143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zx9k\" (UniqueName: \"kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:27.930517 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:27.930474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:28.301217 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.301180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:28.319950 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.319920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:28.322898 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.322875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") pod \"maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:28.401257 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.401221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:28.515416 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.515386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:28.650793 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.650771 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:18:28.652918 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:18:28.652879 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d5661a_262d_4fe3_85ac_f9c5a45a481a.slice/crio-1403d232491dcf14a5299f5d0a17f4a74b42f3c21250dddd0fcc740c281c4c46 WatchSource:0}: Error finding container 1403d232491dcf14a5299f5d0a17f4a74b42f3c21250dddd0fcc740c281c4c46: Status 404 returned error can't find the container with id 1403d232491dcf14a5299f5d0a17f4a74b42f3c21250dddd0fcc740c281c4c46 Jul 18 12:18:28.654633 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:28.654618 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 12:18:29.595477 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:29.595444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" event={"ID":"d5d5661a-262d-4fe3-85ac-f9c5a45a481a","Type":"ContainerStarted","Data":"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3"} Jul 18 12:18:29.595901 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:29.595492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" event={"ID":"d5d5661a-262d-4fe3-85ac-f9c5a45a481a","Type":"ContainerStarted","Data":"1403d232491dcf14a5299f5d0a17f4a74b42f3c21250dddd0fcc740c281c4c46"} Jul 18 12:18:29.595901 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:29.595578 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:29.614886 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:29.614835 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" podStartSLOduration=2.21173178 podStartE2EDuration="2.614818178s" totalImagesPullingTime="403.086398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:18:29.613194681 +0000 UTC m=+1762.103966797" watchObservedRunningTime="2026-07-18 12:18:29.614818178 +0000 UTC m=+1762.105590297" Jul 18 12:18:32.563237 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:32.563203 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:18:35.604785 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:35.604759 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:18:46.542617 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:46.542576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:46.640738 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:46.640671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:49.959664 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:49.959629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:50.060676 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:50.060640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:18:53.059534 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.059495 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr"] Jul 18 12:18:53.063072 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.063049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:53.065285 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.065258 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:18:53.069470 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.069438 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr"] Jul 18 12:18:53.135133 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.135091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-736936a8-bcgwr\" (UID: \"5da268ae-5132-4443-b7b5-9c77ee10a008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:53.236037 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.235997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-736936a8-bcgwr\" (UID: \"5da268ae-5132-4443-b7b5-9c77ee10a008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:53.236605 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.236585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-736936a8-bcgwr\" (UID: \"5da268ae-5132-4443-b7b5-9c77ee10a008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:53.375685 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.375595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:53.512908 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.512882 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr"] Jul 18 12:18:53.515170 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:18:53.515145 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da268ae_5132_4443_b7b5_9c77ee10a008.slice/crio-390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50 WatchSource:0}: Error finding container 390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50: Status 404 returned error can't find the container with id 390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50 Jul 18 12:18:53.705649 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:53.705575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" event={"ID":"5da268ae-5132-4443-b7b5-9c77ee10a008","Type":"ContainerStarted","Data":"390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50"} Jul 18 12:18:54.711392 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:54.711353 2573 generic.go:356] "Generic (PLEG): container finished" podID="5da268ae-5132-4443-b7b5-9c77ee10a008" containerID="31c2bdefcf67f1441b9406f9cb1f48feba00ac9be32289af00a0b1c15ee4b1ff" exitCode=0 Jul 18 12:18:54.711777 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:54.711433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" event={"ID":"5da268ae-5132-4443-b7b5-9c77ee10a008","Type":"ContainerDied","Data":"31c2bdefcf67f1441b9406f9cb1f48feba00ac9be32289af00a0b1c15ee4b1ff"} Jul 18 12:18:55.841148 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:55.841126 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:55.856682 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:55.856652 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca\") pod \"5da268ae-5132-4443-b7b5-9c77ee10a008\" (UID: \"5da268ae-5132-4443-b7b5-9c77ee10a008\") " Jul 18 12:18:55.857068 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:55.857040 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5da268ae-5132-4443-b7b5-9c77ee10a008" (UID: "5da268ae-5132-4443-b7b5-9c77ee10a008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:18:55.957358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:55.957321 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da268ae-5132-4443-b7b5-9c77ee10a008-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:18:56.720805 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:56.720775 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" Jul 18 12:18:56.720978 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:56.720815 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr" event={"ID":"5da268ae-5132-4443-b7b5-9c77ee10a008","Type":"ContainerDied","Data":"390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50"} Jul 18 12:18:56.720978 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:18:56.720861 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="390fa9f2c6fa1493c65b137ecb88165c34598bfdf8750bbb3cdca2db9f6bfc50" Jul 18 12:19:03.466193 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.466109 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:19:03.466642 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.466388 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" podUID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" containerName="maas-api" containerID="cri-o://acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3" gracePeriod=30 Jul 18 12:19:03.500219 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.500189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:03.718134 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.718079 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:19:03.757074 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.757033 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" containerID="acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3" exitCode=0 Jul 18 12:19:03.757245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.757117 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" event={"ID":"d5d5661a-262d-4fe3-85ac-f9c5a45a481a","Type":"ContainerDied","Data":"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3"} Jul 18 12:19:03.757245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.757205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" event={"ID":"d5d5661a-262d-4fe3-85ac-f9c5a45a481a","Type":"ContainerDied","Data":"1403d232491dcf14a5299f5d0a17f4a74b42f3c21250dddd0fcc740c281c4c46"} Jul 18 12:19:03.757245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.757243 2573 scope.go:117] "RemoveContainer" containerID="acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3" Jul 18 12:19:03.757457 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.757440 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj" Jul 18 12:19:03.769681 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.769658 2573 scope.go:117] "RemoveContainer" containerID="acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3" Jul 18 12:19:03.770013 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:03.769991 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3\": container with ID starting with acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3 not found: ID does not exist" containerID="acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3" Jul 18 12:19:03.770095 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.770020 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3"} err="failed to get container status \"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3\": rpc error: code = NotFound desc = could not find container \"acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3\": container with ID starting with acd9d583fcb7ab6f75106aa323ee8d57ab5e689550bcc8a21d8515f8f36bdbb3 not found: ID does not exist" Jul 18 12:19:03.832760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.832702 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zx9k\" (UniqueName: \"kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k\") pod \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " Jul 18 12:19:03.832960 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.832772 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca\") pod \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " Jul 18 12:19:03.832960 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.832897 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") pod \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\" (UID: \"d5d5661a-262d-4fe3-85ac-f9c5a45a481a\") " Jul 18 12:19:03.833140 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.833116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5d5661a-262d-4fe3-85ac-f9c5a45a481a" (UID: "d5d5661a-262d-4fe3-85ac-f9c5a45a481a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:19:03.835120 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.835088 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5d5661a-262d-4fe3-85ac-f9c5a45a481a" (UID: "d5d5661a-262d-4fe3-85ac-f9c5a45a481a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:19:03.835223 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.835162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k" (OuterVolumeSpecName: "kube-api-access-6zx9k") pod "d5d5661a-262d-4fe3-85ac-f9c5a45a481a" (UID: "d5d5661a-262d-4fe3-85ac-f9c5a45a481a"). InnerVolumeSpecName "kube-api-access-6zx9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:03.934351 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.934315 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zx9k\" (UniqueName: \"kubernetes.io/projected/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-kube-api-access-6zx9k\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:03.934351 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.934343 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:03.934351 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:03.934356 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d5661a-262d-4fe3-85ac-f9c5a45a481a-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:04.072772 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:04.072741 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:19:04.076444 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:04.076421 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-736936a8-6f54cc47f5-hh7nj"] Jul 18 12:19:05.955223 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:05.955189 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" path="/var/lib/kubelet/pods/d5d5661a-262d-4fe3-85ac-f9c5a45a481a/volumes" Jul 18 12:19:06.366416 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:06.366382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:19:07.978660 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:07.978634 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:19:07.988029 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:07.988012 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:19:08.188205 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.188165 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr"] Jul 18 12:19:08.191129 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.191099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:08.193308 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.193289 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-736936a8-bcgwr"] Jul 18 12:19:08.345817 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.345785 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:08.574125 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.574098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:08.849807 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.849769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:08.948631 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:08.948604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:09.047929 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:09.047888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:09.103947 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:09.103857 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:09.370409 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:09.370329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:09.955154 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:09.955114 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da268ae-5132-4443-b7b5-9c77ee10a008" path="/var/lib/kubelet/pods/5da268ae-5132-4443-b7b5-9c77ee10a008/volumes" Jul 18 12:19:10.041777 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:10.041687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:11.879223 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.879136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:11.959467 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959434 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:11.959815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959801 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" containerName="maas-api" Jul 18 12:19:11.959867 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959817 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" containerName="maas-api" Jul 18 12:19:11.959867 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959835 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da268ae-5132-4443-b7b5-9c77ee10a008" containerName="revoke-keys" Jul 18 12:19:11.959867 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959841 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da268ae-5132-4443-b7b5-9c77ee10a008" containerName="revoke-keys" Jul 18 12:19:11.959959 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959902 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d5661a-262d-4fe3-85ac-f9c5a45a481a" containerName="maas-api" Jul 18 12:19:11.959959 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.959910 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da268ae-5132-4443-b7b5-9c77ee10a008" containerName="revoke-keys" Jul 18 12:19:11.963955 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.963936 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:11.966040 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.966023 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f1cd424d-serving-cert\"" Jul 18 12:19:11.971660 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:11.971554 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:12.057272 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.057241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:12.100649 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.100609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.100838 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.100656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9psm\" (UniqueName: \"kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.100838 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.100770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.201658 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.201568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.201883 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.201672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.201883 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:12.201723 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-serving-cert: secret "maas-api-e2e-migrate-f1cd424d-serving-cert" not found Jul 18 12:19:12.201883 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:12.201827 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls podName:588be961-0b6c-42b8-a8fa-e199710309ca nodeName:}" failed. No retries permitted until 2026-07-18 12:19:12.701803872 +0000 UTC m=+1805.192575981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls") pod "maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" (UID: "588be961-0b6c-42b8-a8fa-e199710309ca") : secret "maas-api-e2e-migrate-f1cd424d-serving-cert" not found Jul 18 12:19:12.201883 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.201701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9psm\" (UniqueName: \"kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.202429 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.202408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.210516 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.210495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9psm\" (UniqueName: \"kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.651884 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.651846 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:12.706022 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.705979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:12.706172 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:12.706132 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-serving-cert: secret "maas-api-e2e-migrate-f1cd424d-serving-cert" not found Jul 18 12:19:12.706209 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:12.706197 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls podName:588be961-0b6c-42b8-a8fa-e199710309ca nodeName:}" failed. No retries permitted until 2026-07-18 12:19:13.706182633 +0000 UTC m=+1806.196954730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls") pod "maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" (UID: "588be961-0b6c-42b8-a8fa-e199710309ca") : secret "maas-api-e2e-migrate-f1cd424d-serving-cert" not found Jul 18 12:19:12.752046 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.752012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:12.852568 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:12.852531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:13.715190 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:13.715149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:13.717692 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:13.717665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:13.775835 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:13.775807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:13.905615 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:13.905582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:13.909277 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:19:13.909249 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588be961_0b6c_42b8_a8fa_e199710309ca.slice/crio-7244637b7ad0580f10ab7a19373cf6a1b63fed6b0a2a945994e9a27876837984 WatchSource:0}: Error finding container 7244637b7ad0580f10ab7a19373cf6a1b63fed6b0a2a945994e9a27876837984: Status 404 returned error can't find the container with id 7244637b7ad0580f10ab7a19373cf6a1b63fed6b0a2a945994e9a27876837984 Jul 18 12:19:14.808719 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:14.808675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" event={"ID":"588be961-0b6c-42b8-a8fa-e199710309ca","Type":"ContainerStarted","Data":"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e"} Jul 18 12:19:14.809284 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:14.808796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" event={"ID":"588be961-0b6c-42b8-a8fa-e199710309ca","Type":"ContainerStarted","Data":"7244637b7ad0580f10ab7a19373cf6a1b63fed6b0a2a945994e9a27876837984"} Jul 18 12:19:14.826681 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:14.826633 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" podStartSLOduration=3.520561125 podStartE2EDuration="3.826618692s" totalImagesPullingTime="306.057567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:19:14.825319006 +0000 UTC m=+1807.316091123" watchObservedRunningTime="2026-07-18 12:19:14.826618692 +0000 UTC m=+1807.317390808" Jul 18 12:19:15.813047 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:15.813010 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:17.265417 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:17.265382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:19:21.824535 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:21.824506 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:36.876273 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:36.876240 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr"] Jul 18 12:19:36.881053 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:36.881038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:36.883469 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:36.883450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:19:36.887050 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:36.887022 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr"] Jul 18 12:19:36.920200 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:36.920175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr\" (UID: \"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:37.021030 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.021003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr\" (UID: \"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:37.021562 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.021544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr\" (UID: \"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:37.192808 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.192737 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:37.326226 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.326199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr"] Jul 18 12:19:37.329393 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:19:37.329365 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7fe3dd_4a74_4fb1_a40d_d9c82b22c546.slice/crio-0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4 WatchSource:0}: Error finding container 0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4: Status 404 returned error can't find the container with id 0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4 Jul 18 12:19:37.919078 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.918995 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" event={"ID":"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546","Type":"ContainerStarted","Data":"8a1c67ff1eedbe1a81063e14d57801fd6a1249db05eff211a9f569369bc88595"} Jul 18 12:19:37.919078 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.919045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" event={"ID":"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546","Type":"ContainerStarted","Data":"0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4"} Jul 18 12:19:37.935993 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:37.935953 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" podStartSLOduration=1.9359384899999998 podStartE2EDuration="1.93593849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:19:37.93338598 +0000 UTC m=+1830.424158094" watchObservedRunningTime="2026-07-18 12:19:37.93593849 +0000 UTC m=+1830.426710658" Jul 18 12:19:38.924823 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:38.924790 2573 generic.go:356] "Generic (PLEG): container finished" podID="bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" containerID="8a1c67ff1eedbe1a81063e14d57801fd6a1249db05eff211a9f569369bc88595" exitCode=0 Jul 18 12:19:38.925219 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:38.924885 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" event={"ID":"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546","Type":"ContainerDied","Data":"8a1c67ff1eedbe1a81063e14d57801fd6a1249db05eff211a9f569369bc88595"} Jul 18 12:19:40.060100 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.060075 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:40.147374 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.147342 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca\") pod \"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546\" (UID: \"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546\") " Jul 18 12:19:40.147761 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.147743 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" (UID: "bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:19:40.248922 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.248900 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:40.934625 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.934583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" event={"ID":"bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546","Type":"ContainerDied","Data":"0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4"} Jul 18 12:19:40.934625 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.934617 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr" Jul 18 12:19:40.934858 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:40.934646 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0913f85b570f4d7a521de7e6261740dd61cd046e19daa4f48f45c41c5e1620a4" Jul 18 12:19:47.357474 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.357439 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:47.357929 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.357645 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" podUID="588be961-0b6c-42b8-a8fa-e199710309ca" containerName="maas-api" containerID="cri-o://cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e" gracePeriod=30 Jul 18 12:19:47.381823 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.381791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:47.607805 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.607749 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:47.705930 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.705899 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca\") pod \"588be961-0b6c-42b8-a8fa-e199710309ca\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " Jul 18 12:19:47.706110 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.705937 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") pod \"588be961-0b6c-42b8-a8fa-e199710309ca\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " Jul 18 12:19:47.706110 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.705985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9psm\" (UniqueName: \"kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm\") pod \"588be961-0b6c-42b8-a8fa-e199710309ca\" (UID: \"588be961-0b6c-42b8-a8fa-e199710309ca\") " Jul 18 12:19:47.706247 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.706221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "588be961-0b6c-42b8-a8fa-e199710309ca" (UID: "588be961-0b6c-42b8-a8fa-e199710309ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:19:47.708164 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.708141 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm" (OuterVolumeSpecName: "kube-api-access-s9psm") pod "588be961-0b6c-42b8-a8fa-e199710309ca" (UID: "588be961-0b6c-42b8-a8fa-e199710309ca"). InnerVolumeSpecName "kube-api-access-s9psm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:47.708285 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.708190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "588be961-0b6c-42b8-a8fa-e199710309ca" (UID: "588be961-0b6c-42b8-a8fa-e199710309ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:19:47.807339 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.807305 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/588be961-0b6c-42b8-a8fa-e199710309ca-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:47.807339 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.807333 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/588be961-0b6c-42b8-a8fa-e199710309ca-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:47.807339 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.807341 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9psm\" (UniqueName: \"kubernetes.io/projected/588be961-0b6c-42b8-a8fa-e199710309ca-kube-api-access-s9psm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:19:47.966943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.966855 2573 generic.go:356] "Generic (PLEG): container finished" podID="588be961-0b6c-42b8-a8fa-e199710309ca" containerID="cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e" exitCode=0 Jul 18 12:19:47.966943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.966903 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" Jul 18 12:19:47.967122 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.966940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" event={"ID":"588be961-0b6c-42b8-a8fa-e199710309ca","Type":"ContainerDied","Data":"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e"} Jul 18 12:19:47.967122 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.966995 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq" event={"ID":"588be961-0b6c-42b8-a8fa-e199710309ca","Type":"ContainerDied","Data":"7244637b7ad0580f10ab7a19373cf6a1b63fed6b0a2a945994e9a27876837984"} Jul 18 12:19:47.967122 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.967026 2573 scope.go:117] "RemoveContainer" containerID="cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e" Jul 18 12:19:47.976260 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.976250 2573 scope.go:117] "RemoveContainer" containerID="cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e" Jul 18 12:19:47.976532 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:19:47.976508 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e\": container with ID starting with cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e not found: ID does not exist" containerID="cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e" Jul 18 12:19:47.976603 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.976542 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e"} err="failed to get container status \"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e\": rpc error: code = NotFound desc = could not find container \"cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e\": container with ID starting with cbd7ca16b271a747966211fac4f7e76a3017b5ff6e81aa3791544dd6bcd1bb5e not found: ID does not exist" Jul 18 12:19:47.986726 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.986690 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:47.990537 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:47.990511 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f1cd424d-868f4f57f-5r8hq"] Jul 18 12:19:49.955368 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:49.955338 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588be961-0b6c-42b8-a8fa-e199710309ca" path="/var/lib/kubelet/pods/588be961-0b6c-42b8-a8fa-e199710309ca/volumes" Jul 18 12:19:50.070966 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:50.070930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:19:51.950005 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:51.949971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:51.954849 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:51.954828 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr"] Jul 18 12:19:51.954958 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:51.954858 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f1cd424d-tfkhr"] Jul 18 12:19:52.104301 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:52.104260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:52.338383 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:52.338348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:52.559528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:52.559495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:52.859390 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:52.859355 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:53.027771 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:53.027734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:53.955390 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:53.955358 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" path="/var/lib/kubelet/pods/bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546/volumes" Jul 18 12:19:54.994085 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:54.994048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:55.078405 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.078374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:55.079361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079328 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:19:55.079787 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079765 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" containerName="revoke-keys" Jul 18 12:19:55.079787 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079787 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" containerName="revoke-keys" Jul 18 12:19:55.079956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079803 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588be961-0b6c-42b8-a8fa-e199710309ca" containerName="maas-api" Jul 18 12:19:55.079956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079809 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="588be961-0b6c-42b8-a8fa-e199710309ca" containerName="maas-api" Jul 18 12:19:55.079956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7fe3dd-4a74-4fb1-a40d-d9c82b22c546" containerName="revoke-keys" Jul 18 12:19:55.079956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.079890 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="588be961-0b6c-42b8-a8fa-e199710309ca" containerName="maas-api" Jul 18 12:19:55.084547 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.084531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.087283 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.087232 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3dff9e6d-serving-cert\"" Jul 18 12:19:55.093580 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.093267 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:19:55.147141 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.147106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:55.173151 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.173119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.173304 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.173156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.173304 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.173177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcb84\" (UniqueName: \"kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.273700 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.273665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.273700 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.273701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.273939 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.273743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcb84\" (UniqueName: \"kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.274362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.274343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.276258 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.276238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.288358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.288330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcb84\" (UniqueName: \"kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84\") pod \"maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.356435 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.356392 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:55.398490 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.398454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:55.541550 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.541524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:19:55.543558 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:19:55.543531 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8507ad_2236_4c04_8307_e4627e2f3bb6.slice/crio-270a24cf0a6ba0f9355993f739a88df16b5e0641d10e978aee22ea20367267e6 WatchSource:0}: Error finding container 270a24cf0a6ba0f9355993f739a88df16b5e0641d10e978aee22ea20367267e6: Status 404 returned error can't find the container with id 270a24cf0a6ba0f9355993f739a88df16b5e0641d10e978aee22ea20367267e6 Jul 18 12:19:55.681775 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.681737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:55.779063 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:55.779032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:19:56.016347 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:56.016314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" event={"ID":"fe8507ad-2236-4c04-8307-e4627e2f3bb6","Type":"ContainerStarted","Data":"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856"} Jul 18 12:19:56.016776 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:56.016364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" event={"ID":"fe8507ad-2236-4c04-8307-e4627e2f3bb6","Type":"ContainerStarted","Data":"270a24cf0a6ba0f9355993f739a88df16b5e0641d10e978aee22ea20367267e6"} Jul 18 12:19:56.016776 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:56.016415 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:19:56.036105 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:19:56.036047 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" podStartSLOduration=0.672190646 podStartE2EDuration="1.036032638s" totalImagesPullingTime="363.841992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:19:56.034442311 +0000 UTC m=+1848.525214430" watchObservedRunningTime="2026-07-18 12:19:56.036032638 +0000 UTC m=+1848.526804754" Jul 18 12:20:00.669926 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:00.669894 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:20:02.027306 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:02.027279 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:20:31.231498 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.231462 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:20:31.232397 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.232325 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" podUID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" containerName="maas-api" containerID="cri-o://193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856" gracePeriod=30 Jul 18 12:20:31.270879 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.270848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:31.487428 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.487367 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:20:31.586160 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.586128 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcb84\" (UniqueName: \"kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84\") pod \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " Jul 18 12:20:31.586355 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.586173 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca\") pod \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " Jul 18 12:20:31.586355 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.586207 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls\") pod \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\" (UID: \"fe8507ad-2236-4c04-8307-e4627e2f3bb6\") " Jul 18 12:20:31.586586 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.586564 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe8507ad-2236-4c04-8307-e4627e2f3bb6" (UID: "fe8507ad-2236-4c04-8307-e4627e2f3bb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:20:31.588672 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.588642 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe8507ad-2236-4c04-8307-e4627e2f3bb6" (UID: "fe8507ad-2236-4c04-8307-e4627e2f3bb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:20:31.588672 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.588656 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84" (OuterVolumeSpecName: "kube-api-access-fcb84") pod "fe8507ad-2236-4c04-8307-e4627e2f3bb6" (UID: "fe8507ad-2236-4c04-8307-e4627e2f3bb6"). InnerVolumeSpecName "kube-api-access-fcb84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:20:31.687506 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.687464 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8507ad-2236-4c04-8307-e4627e2f3bb6-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:20:31.687506 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.687503 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcb84\" (UniqueName: \"kubernetes.io/projected/fe8507ad-2236-4c04-8307-e4627e2f3bb6-kube-api-access-fcb84\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:20:31.687726 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:31.687518 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8507ad-2236-4c04-8307-e4627e2f3bb6-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:20:32.182545 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.182510 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" containerID="193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856" exitCode=0 Jul 18 12:20:32.182760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.182560 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" Jul 18 12:20:32.182760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.182595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" event={"ID":"fe8507ad-2236-4c04-8307-e4627e2f3bb6","Type":"ContainerDied","Data":"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856"} Jul 18 12:20:32.182760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.182651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh" event={"ID":"fe8507ad-2236-4c04-8307-e4627e2f3bb6","Type":"ContainerDied","Data":"270a24cf0a6ba0f9355993f739a88df16b5e0641d10e978aee22ea20367267e6"} Jul 18 12:20:32.182760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.182677 2573 scope.go:117] "RemoveContainer" containerID="193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856" Jul 18 12:20:32.197481 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.197454 2573 scope.go:117] "RemoveContainer" containerID="193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856" Jul 18 12:20:32.197846 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:20:32.197820 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856\": container with ID starting with 193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856 not found: ID does not exist" containerID="193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856" Jul 18 12:20:32.197921 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.197859 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856"} err="failed to get container status \"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856\": rpc error: code = NotFound desc = could not find container \"193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856\": container with ID starting with 193bb4f27b9a9f37412b2e48c511ac4ef0d93a037cec289a3afaf22551ba9856 not found: ID does not exist" Jul 18 12:20:32.200375 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.200351 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:20:32.204507 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:32.204483 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dff9e6d-f7bdbf647-fckfh"] Jul 18 12:20:33.955486 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:33.955450 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" path="/var/lib/kubelet/pods/fe8507ad-2236-4c04-8307-e4627e2f3bb6/volumes" Jul 18 12:20:34.275118 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:34.275085 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:20:35.853505 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:35.853455 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:36.991943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:36.991912 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:37.190345 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:37.190306 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:37.399984 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:37.399948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:37.826310 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:37.826279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:37.948723 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:37.948675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:40.256157 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.256116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:40.360767 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.360735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:40.466198 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.466170 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:20:40.466531 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.466519 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" containerName="maas-api" Jul 18 12:20:40.466588 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.466533 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" containerName="maas-api" Jul 18 12:20:40.466624 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.466591 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8507ad-2236-4c04-8307-e4627e2f3bb6" containerName="maas-api" Jul 18 12:20:40.469393 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.469378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.472097 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.472072 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c5db2665-serving-cert\"" Jul 18 12:20:40.494767 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.494741 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:20:40.538858 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.538771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:40.562066 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.562037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgz2\" (UniqueName: \"kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.562245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.562105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.562245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.562148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.662581 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.662544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.662812 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.662591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.662812 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.662626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgz2\" (UniqueName: \"kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.662812 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:20:40.662694 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-serving-cert: secret "maas-api-e2e-derive-c5db2665-serving-cert" not found Jul 18 12:20:40.662812 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:20:40.662798 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls podName:60b0c678-b870-412e-a7b3-a2ec8d0a38bf nodeName:}" failed. No retries permitted until 2026-07-18 12:20:41.162780378 +0000 UTC m=+1893.653552473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls") pod "maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" (UID: "60b0c678-b870-412e-a7b3-a2ec8d0a38bf") : secret "maas-api-e2e-derive-c5db2665-serving-cert" not found Jul 18 12:20:40.663351 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.663328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:40.675846 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:40.675821 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgz2\" (UniqueName: \"kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:41.043847 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:41.043817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:41.144463 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:41.144425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:41.167560 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:41.167525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:41.167748 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:20:41.167661 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-serving-cert: secret "maas-api-e2e-derive-c5db2665-serving-cert" not found Jul 18 12:20:41.167816 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:20:41.167771 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls podName:60b0c678-b870-412e-a7b3-a2ec8d0a38bf nodeName:}" failed. No retries permitted until 2026-07-18 12:20:42.167756195 +0000 UTC m=+1894.658528293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls") pod "maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" (UID: "60b0c678-b870-412e-a7b3-a2ec8d0a38bf") : secret "maas-api-e2e-derive-c5db2665-serving-cert" not found Jul 18 12:20:41.383395 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:41.383305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:20:42.177077 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:42.177045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:42.179514 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:42.179485 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"maas-api-e2e-derive-c5db2665-f776dbc98-5rj84\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:42.281789 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:42.281758 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:42.420000 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:42.419976 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:20:42.421746 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:20:42.421702 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b0c678_b870_412e_a7b3_a2ec8d0a38bf.slice/crio-f0342b7742a24fcbca90489a02531a37d8e27c894998107590fb9f2b782c22ee WatchSource:0}: Error finding container f0342b7742a24fcbca90489a02531a37d8e27c894998107590fb9f2b782c22ee: Status 404 returned error can't find the container with id f0342b7742a24fcbca90489a02531a37d8e27c894998107590fb9f2b782c22ee Jul 18 12:20:43.239433 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:43.239338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" event={"ID":"60b0c678-b870-412e-a7b3-a2ec8d0a38bf","Type":"ContainerStarted","Data":"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02"} Jul 18 12:20:43.239611 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:43.239444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" event={"ID":"60b0c678-b870-412e-a7b3-a2ec8d0a38bf","Type":"ContainerStarted","Data":"f0342b7742a24fcbca90489a02531a37d8e27c894998107590fb9f2b782c22ee"} Jul 18 12:20:43.240522 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:43.240495 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:20:43.260459 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:43.260405 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" podStartSLOduration=2.792410087 podStartE2EDuration="3.260391189s" totalImagesPullingTime="467.981102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:20:43.258321514 +0000 UTC m=+1895.749093635" watchObservedRunningTime="2026-07-18 12:20:43.260391189 +0000 UTC m=+1895.751163305" Jul 18 12:20:44.777341 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:44.777296 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:20:50.254571 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:20:50.254540 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:21:05.126362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.126319 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv"] Jul 18 12:21:05.130287 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.130263 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:05.132747 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.132702 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:21:05.137857 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.137629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv"] Jul 18 12:21:05.282460 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.282423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv\" (UID: \"06a58b14-272a-4ba3-88f7-f3789dbeb28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:05.383221 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.383134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv\" (UID: \"06a58b14-272a-4ba3-88f7-f3789dbeb28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:05.383817 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.383800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv\" (UID: \"06a58b14-272a-4ba3-88f7-f3789dbeb28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:05.442198 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.442164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:05.581304 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:05.581282 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv"] Jul 18 12:21:05.583306 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:21:05.583277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a58b14_272a_4ba3_88f7_f3789dbeb28d.slice/crio-2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e WatchSource:0}: Error finding container 2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e: Status 404 returned error can't find the container with id 2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e Jul 18 12:21:06.345039 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:06.345001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" event={"ID":"06a58b14-272a-4ba3-88f7-f3789dbeb28d","Type":"ContainerStarted","Data":"d5e0e026a6f1785afc548f6e6bbede722e6cb0fd0bf230dfcf821fcfb0b77176"} Jul 18 12:21:06.345412 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:06.345063 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" event={"ID":"06a58b14-272a-4ba3-88f7-f3789dbeb28d","Type":"ContainerStarted","Data":"2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e"} Jul 18 12:21:06.362876 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:06.362827 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" podStartSLOduration=1.362813431 podStartE2EDuration="1.362813431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:21:06.36055774 +0000 UTC m=+1918.851329856" watchObservedRunningTime="2026-07-18 12:21:06.362813431 +0000 UTC m=+1918.853585588" Jul 18 12:21:07.350991 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:07.350959 2573 generic.go:356] "Generic (PLEG): container finished" podID="06a58b14-272a-4ba3-88f7-f3789dbeb28d" containerID="d5e0e026a6f1785afc548f6e6bbede722e6cb0fd0bf230dfcf821fcfb0b77176" exitCode=0 Jul 18 12:21:07.351448 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:07.351057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" event={"ID":"06a58b14-272a-4ba3-88f7-f3789dbeb28d","Type":"ContainerDied","Data":"d5e0e026a6f1785afc548f6e6bbede722e6cb0fd0bf230dfcf821fcfb0b77176"} Jul 18 12:21:08.489930 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:08.489909 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:08.611643 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:08.611554 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca\") pod \"06a58b14-272a-4ba3-88f7-f3789dbeb28d\" (UID: \"06a58b14-272a-4ba3-88f7-f3789dbeb28d\") " Jul 18 12:21:08.611956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:08.611936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06a58b14-272a-4ba3-88f7-f3789dbeb28d" (UID: "06a58b14-272a-4ba3-88f7-f3789dbeb28d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:21:08.712542 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:08.712515 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a58b14-272a-4ba3-88f7-f3789dbeb28d-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:21:09.362888 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:09.362846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" event={"ID":"06a58b14-272a-4ba3-88f7-f3789dbeb28d","Type":"ContainerDied","Data":"2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e"} Jul 18 12:21:09.363106 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:09.362954 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ccf8c27611cc20fff3fba1745380826aefe9e6cb993fe345a255bf5a2efb52e" Jul 18 12:21:09.363106 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:09.363033 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv" Jul 18 12:21:15.675170 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:15.675140 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:21:15.675624 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:15.675327 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" podUID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" containerName="maas-api" containerID="cri-o://0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02" gracePeriod=30 Jul 18 12:21:15.702027 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:15.702003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:15.928209 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:15.928154 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:21:16.071497 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.071468 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") pod \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " Jul 18 12:21:16.071672 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.071509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvgz2\" (UniqueName: \"kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2\") pod \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " Jul 18 12:21:16.071672 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.071656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca\") pod \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\" (UID: \"60b0c678-b870-412e-a7b3-a2ec8d0a38bf\") " Jul 18 12:21:16.072024 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.071985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60b0c678-b870-412e-a7b3-a2ec8d0a38bf" (UID: "60b0c678-b870-412e-a7b3-a2ec8d0a38bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:21:16.073799 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.073778 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2" (OuterVolumeSpecName: "kube-api-access-cvgz2") pod "60b0c678-b870-412e-a7b3-a2ec8d0a38bf" (UID: "60b0c678-b870-412e-a7b3-a2ec8d0a38bf"). InnerVolumeSpecName "kube-api-access-cvgz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:21:16.073912 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.073899 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60b0c678-b870-412e-a7b3-a2ec8d0a38bf" (UID: "60b0c678-b870-412e-a7b3-a2ec8d0a38bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:21:16.172842 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.172811 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:21:16.172842 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.172835 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:21:16.172842 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.172844 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgz2\" (UniqueName: \"kubernetes.io/projected/60b0c678-b870-412e-a7b3-a2ec8d0a38bf-kube-api-access-cvgz2\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:21:16.394867 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.394834 2573 generic.go:356] "Generic (PLEG): container finished" podID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" containerID="0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02" exitCode=0 Jul 18 12:21:16.395060 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.394886 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" Jul 18 12:21:16.395060 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.394928 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" event={"ID":"60b0c678-b870-412e-a7b3-a2ec8d0a38bf","Type":"ContainerDied","Data":"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02"} Jul 18 12:21:16.395060 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.394985 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84" event={"ID":"60b0c678-b870-412e-a7b3-a2ec8d0a38bf","Type":"ContainerDied","Data":"f0342b7742a24fcbca90489a02531a37d8e27c894998107590fb9f2b782c22ee"} Jul 18 12:21:16.395060 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.395014 2573 scope.go:117] "RemoveContainer" containerID="0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02" Jul 18 12:21:16.406480 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.406458 2573 scope.go:117] "RemoveContainer" containerID="0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02" Jul 18 12:21:16.406811 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:21:16.406788 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02\": container with ID starting with 0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02 not found: ID does not exist" containerID="0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02" Jul 18 12:21:16.406907 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.406821 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02"} err="failed to get container status \"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02\": rpc error: code = NotFound desc = could not find container \"0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02\": container with ID starting with 0e14438e2a1abcc79b48610377005cb7153af7a1d7701857ad6cfb2f3708cf02 not found: ID does not exist" Jul 18 12:21:16.416790 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.416764 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:21:16.420870 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:16.420852 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c5db2665-f776dbc98-5rj84"] Jul 18 12:21:17.955390 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:17.955359 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" path="/var/lib/kubelet/pods/60b0c678-b870-412e-a7b3-a2ec8d0a38bf/volumes" Jul 18 12:21:18.477723 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:18.477683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:21:20.214286 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:20.214252 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv"] Jul 18 12:21:20.217832 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:20.217807 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5db2665-gk9zv"] Jul 18 12:21:20.240462 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:20.240423 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:20.390956 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:20.390898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:20.606973 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:20.606933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:21.080420 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:21.080386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:21.180088 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:21.180056 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:21.280165 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:21.280104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:21.955032 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:21.955000 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06a58b14-272a-4ba3-88f7-f3789dbeb28d" path="/var/lib/kubelet/pods/06a58b14-272a-4ba3-88f7-f3789dbeb28d/volumes" Jul 18 12:21:22.670141 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:22.670113 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:29.354530 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:29.354487 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:29.459546 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:29.459517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:29.555503 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:29.555469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:29.658546 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:29.658458 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:21:35.147623 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:21:35.147579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:01.122618 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:01.122532 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:07.465331 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:07.465272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:07.567081 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:07.567040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:07.662873 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:07.662834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:23.565093 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:23.565060 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:29.225175 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:29.225137 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:29.554473 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:29.554440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:29.627416 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:29.627381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:29.718027 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:29.717996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:29.811277 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:29.811194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:40.274659 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:40.274621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:41.052728 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:41.052683 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:47.746201 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:47.746158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:47.849325 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:47.849288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:47.946731 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:47.946662 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:48.047390 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:48.047303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:48.803772 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:48.803732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:48.892997 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:48.892960 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:48.992732 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:48.992684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:50.626561 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:50.626528 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:51.082879 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:51.082844 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:22:56.468866 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:56.468833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:56.540378 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:56.540345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:22:57.136567 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:22:57.136539 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:02.383262 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:02.383231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:23:04.234110 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:04.234073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:04.332029 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:04.331998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:04.472734 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:04.472671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:04.795968 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:04.795933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:05.092349 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:05.092260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:07.667046 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.667013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:07.854130 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854099 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:07.854525 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854508 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a58b14-272a-4ba3-88f7-f3789dbeb28d" containerName="revoke-keys" Jul 18 12:23:07.854610 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854527 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a58b14-272a-4ba3-88f7-f3789dbeb28d" containerName="revoke-keys" Jul 18 12:23:07.854610 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854546 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" containerName="maas-api" Jul 18 12:23:07.854610 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854554 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" containerName="maas-api" Jul 18 12:23:07.854798 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854663 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06a58b14-272a-4ba3-88f7-f3789dbeb28d" containerName="revoke-keys" Jul 18 12:23:07.854798 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.854680 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b0c678-b870-412e-a7b3-a2ec8d0a38bf" containerName="maas-api" Jul 18 12:23:07.857815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.857790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:07.859815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.859783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:07.863815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.863794 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a9c69c8d-serving-cert\"" Jul 18 12:23:07.870182 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.870158 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:07.928114 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.928016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:07.928114 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.928104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:07.928302 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.928166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:07.956919 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:07.956880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:08.029135 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.029091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.029347 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.029142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.029347 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.029208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.029347 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:23:08.029260 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-serving-cert: secret "maas-api-e2e-mt-a9c69c8d-serving-cert" not found Jul 18 12:23:08.029347 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:23:08.029338 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls podName:7c486a5d-00f1-489d-ac6e-9e1dd2bec203 nodeName:}" failed. No retries permitted until 2026-07-18 12:23:08.529317623 +0000 UTC m=+2041.020089737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls") pod "maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" (UID: "7c486a5d-00f1-489d-ac6e-9e1dd2bec203") : secret "maas-api-e2e-mt-a9c69c8d-serving-cert" not found Jul 18 12:23:08.029855 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.029837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.038053 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.038022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.366958 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.366921 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:08.535004 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.534969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.537632 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.537609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") pod \"maas-api-e2e-mt-a9c69c8d-6689964888-94nfv\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.570768 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.570734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:08.672739 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.672631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:08.768818 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.768780 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:08.907876 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.907845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:08.910098 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:23:08.910074 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c486a5d_00f1_489d_ac6e_9e1dd2bec203.slice/crio-b82012566b7eee977bf712c51fa7e9dbf792fcb7668f78cbbe7e70d7bf3c3e8a WatchSource:0}: Error finding container b82012566b7eee977bf712c51fa7e9dbf792fcb7668f78cbbe7e70d7bf3c3e8a: Status 404 returned error can't find the container with id b82012566b7eee977bf712c51fa7e9dbf792fcb7668f78cbbe7e70d7bf3c3e8a Jul 18 12:23:08.921527 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:08.921494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" event={"ID":"7c486a5d-00f1-489d-ac6e-9e1dd2bec203","Type":"ContainerStarted","Data":"b82012566b7eee977bf712c51fa7e9dbf792fcb7668f78cbbe7e70d7bf3c3e8a"} Jul 18 12:23:09.927649 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:09.927605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" event={"ID":"7c486a5d-00f1-489d-ac6e-9e1dd2bec203","Type":"ContainerStarted","Data":"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972"} Jul 18 12:23:09.928171 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:09.927742 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:09.946463 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:09.946405 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" podStartSLOduration=2.507687518 podStartE2EDuration="2.946389153s" totalImagesPullingTime="438.701635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:23:09.945442066 +0000 UTC m=+2042.436214182" watchObservedRunningTime="2026-07-18 12:23:09.946389153 +0000 UTC m=+2042.437161269" Jul 18 12:23:12.129333 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:12.129293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:12.765372 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:12.765342 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:12.862031 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:12.861993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:12.980406 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:12.980371 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:23:13.155393 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:13.155310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:15.937976 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:15.937950 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:19.134860 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:19.134822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:19.237139 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:19.237105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:29.934542 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:29.934488 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:29.935052 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:29.934799 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" podUID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" containerName="maas-api" containerID="cri-o://591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972" gracePeriod=30 Jul 18 12:23:29.968023 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:29.967991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:30.183545 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.183523 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:30.334457 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.334429 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") pod \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " Jul 18 12:23:30.334647 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.334547 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4\") pod \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " Jul 18 12:23:30.334647 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.334570 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca\") pod \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\" (UID: \"7c486a5d-00f1-489d-ac6e-9e1dd2bec203\") " Jul 18 12:23:30.334981 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.334958 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c486a5d-00f1-489d-ac6e-9e1dd2bec203" (UID: "7c486a5d-00f1-489d-ac6e-9e1dd2bec203"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:23:30.336756 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.336726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4" (OuterVolumeSpecName: "kube-api-access-lrvr4") pod "7c486a5d-00f1-489d-ac6e-9e1dd2bec203" (UID: "7c486a5d-00f1-489d-ac6e-9e1dd2bec203"). InnerVolumeSpecName "kube-api-access-lrvr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:23:30.336756 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.336724 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c486a5d-00f1-489d-ac6e-9e1dd2bec203" (UID: "7c486a5d-00f1-489d-ac6e-9e1dd2bec203"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:23:30.436066 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.436032 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-kube-api-access-lrvr4\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:23:30.436066 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.436062 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:23:30.436066 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:30.436072 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c486a5d-00f1-489d-ac6e-9e1dd2bec203-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:23:31.024200 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.024168 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" containerID="591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972" exitCode=0 Jul 18 12:23:31.024633 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.024209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" Jul 18 12:23:31.024633 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.024234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" event={"ID":"7c486a5d-00f1-489d-ac6e-9e1dd2bec203","Type":"ContainerDied","Data":"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972"} Jul 18 12:23:31.024633 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.024283 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv" event={"ID":"7c486a5d-00f1-489d-ac6e-9e1dd2bec203","Type":"ContainerDied","Data":"b82012566b7eee977bf712c51fa7e9dbf792fcb7668f78cbbe7e70d7bf3c3e8a"} Jul 18 12:23:31.024633 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.024308 2573 scope.go:117] "RemoveContainer" containerID="591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972" Jul 18 12:23:31.032987 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.032970 2573 scope.go:117] "RemoveContainer" containerID="591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972" Jul 18 12:23:31.033248 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:23:31.033229 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972\": container with ID starting with 591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972 not found: ID does not exist" containerID="591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972" Jul 18 12:23:31.033309 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.033261 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972"} err="failed to get container status \"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972\": rpc error: code = NotFound desc = could not find container \"591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972\": container with ID starting with 591d6ba6d5f95ee0ac15a45e8674474e8cf525ba548c9b27b186190b2bf14972 not found: ID does not exist" Jul 18 12:23:31.045755 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.045724 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:31.049556 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.049534 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a9c69c8d-6689964888-94nfv"] Jul 18 12:23:31.955092 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:31.955060 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" path="/var/lib/kubelet/pods/7c486a5d-00f1-489d-ac6e-9e1dd2bec203/volumes" Jul 18 12:23:33.088680 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:33.088649 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:23:34.562594 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:34.562561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:34.935906 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:34.935809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.042195 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.042154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.108981 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.108950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.299989 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.299908 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.509247 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.509210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.623697 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.623597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.717428 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.717383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:41.833681 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:41.833640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:47.417120 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:47.417082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:47.517651 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:47.517617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:48.078584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:48.078555 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:48.167757 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:48.167694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:48.266305 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:48.266268 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:48.507874 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:48.507845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:54.658458 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:54.658425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:23:54.764851 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:23:54.764822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:01.511217 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:01.511179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:01.895890 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:01.895771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:02.121596 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:02.121553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:04.839866 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.839827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:04.916952 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.916915 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:24:04.917510 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.917495 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" containerName="maas-api" Jul 18 12:24:04.917557 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.917515 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" containerName="maas-api" Jul 18 12:24:04.917604 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.917594 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c486a5d-00f1-489d-ac6e-9e1dd2bec203" containerName="maas-api" Jul 18 12:24:04.920983 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.920967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:04.924002 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.923977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-07bfbcac-serving-cert\"" Jul 18 12:24:04.935767 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.935741 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:24:04.943292 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:04.943262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:05.023022 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.022968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:05.040121 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.040090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.040322 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.040145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.040322 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.040220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxfc\" (UniqueName: \"kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.141210 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.141121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.141210 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.141166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxfc\" (UniqueName: \"kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.141436 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.141215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.141436 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:24:05.141323 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-serving-cert: secret "maas-api-e2e-mt-07bfbcac-serving-cert" not found Jul 18 12:24:05.141436 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:24:05.141395 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls podName:0444c3d5-a18a-4c5f-90c0-30eebb217127 nodeName:}" failed. No retries permitted until 2026-07-18 12:24:05.641374573 +0000 UTC m=+2098.132146689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls") pod "maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" (UID: "0444c3d5-a18a-4c5f-90c0-30eebb217127") : secret "maas-api-e2e-mt-07bfbcac-serving-cert" not found Jul 18 12:24:05.141800 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.141779 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.151263 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.151235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxfc\" (UniqueName: \"kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.264894 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.264862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:05.639362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.639328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:05.645949 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.645920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.648434 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.648408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") pod \"maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.740024 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.739996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:05.831655 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.831613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:05.970192 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.970168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:24:05.972010 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:24:05.971981 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0444c3d5_a18a_4c5f_90c0_30eebb217127.slice/crio-5ccfbf4c7a0be5b244c2d1752c9855539e5847a4dac0a03d575d9741a95d8e2f WatchSource:0}: Error finding container 5ccfbf4c7a0be5b244c2d1752c9855539e5847a4dac0a03d575d9741a95d8e2f: Status 404 returned error can't find the container with id 5ccfbf4c7a0be5b244c2d1752c9855539e5847a4dac0a03d575d9741a95d8e2f Jul 18 12:24:05.973299 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:05.973277 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 12:24:06.197124 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:06.197038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" event={"ID":"0444c3d5-a18a-4c5f-90c0-30eebb217127","Type":"ContainerStarted","Data":"5ccfbf4c7a0be5b244c2d1752c9855539e5847a4dac0a03d575d9741a95d8e2f"} Jul 18 12:24:07.203868 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:07.203825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" event={"ID":"0444c3d5-a18a-4c5f-90c0-30eebb217127","Type":"ContainerStarted","Data":"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510"} Jul 18 12:24:07.204319 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:07.203903 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:07.225909 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:07.225856 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" podStartSLOduration=2.901605966 podStartE2EDuration="3.225839575s" totalImagesPullingTime="324.233609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:24:07.223751447 +0000 UTC m=+2099.714523566" watchObservedRunningTime="2026-07-18 12:24:07.225839575 +0000 UTC m=+2099.716611692" Jul 18 12:24:08.006411 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:08.006378 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:24:08.021532 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:08.021504 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:24:09.175502 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:09.175467 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:24:09.311904 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:09.311868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:13.213754 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:13.213690 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:24:18.669187 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:18.669153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:24:21.874595 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:21.874557 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:24:44.081438 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:24:44.081403 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:25:02.065775 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:02.065735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:25:03.061363 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.061322 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:25:03.062256 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.061593 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerName="maas-api" containerID="cri-o://46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510" gracePeriod=30 Jul 18 12:25:03.089135 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.089094 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:03.209744 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.209684 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Jul 18 12:25:03.317110 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.317048 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:25:03.455702 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.455665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca\") pod \"0444c3d5-a18a-4c5f-90c0-30eebb217127\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " Jul 18 12:25:03.455895 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.455703 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njxfc\" (UniqueName: \"kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc\") pod \"0444c3d5-a18a-4c5f-90c0-30eebb217127\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " Jul 18 12:25:03.455895 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.455866 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") pod \"0444c3d5-a18a-4c5f-90c0-30eebb217127\" (UID: \"0444c3d5-a18a-4c5f-90c0-30eebb217127\") " Jul 18 12:25:03.456065 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.456045 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0444c3d5-a18a-4c5f-90c0-30eebb217127" (UID: "0444c3d5-a18a-4c5f-90c0-30eebb217127"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:25:03.456120 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.456107 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0444c3d5-a18a-4c5f-90c0-30eebb217127-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:25:03.457996 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.457969 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc" (OuterVolumeSpecName: "kube-api-access-njxfc") pod "0444c3d5-a18a-4c5f-90c0-30eebb217127" (UID: "0444c3d5-a18a-4c5f-90c0-30eebb217127"). InnerVolumeSpecName "kube-api-access-njxfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:25:03.458062 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.457989 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0444c3d5-a18a-4c5f-90c0-30eebb217127" (UID: "0444c3d5-a18a-4c5f-90c0-30eebb217127"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:25:03.474326 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.474298 2573 generic.go:356] "Generic (PLEG): container finished" podID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerID="46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510" exitCode=0 Jul 18 12:25:03.474470 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.474341 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" Jul 18 12:25:03.474470 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.474356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" event={"ID":"0444c3d5-a18a-4c5f-90c0-30eebb217127","Type":"ContainerDied","Data":"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510"} Jul 18 12:25:03.474470 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.474396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4" event={"ID":"0444c3d5-a18a-4c5f-90c0-30eebb217127","Type":"ContainerDied","Data":"5ccfbf4c7a0be5b244c2d1752c9855539e5847a4dac0a03d575d9741a95d8e2f"} Jul 18 12:25:03.474470 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.474417 2573 scope.go:117] "RemoveContainer" containerID="46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510" Jul 18 12:25:03.483083 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.483064 2573 scope.go:117] "RemoveContainer" containerID="46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510" Jul 18 12:25:03.483352 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:25:03.483333 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510\": container with ID starting with 46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510 not found: ID does not exist" containerID="46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510" Jul 18 12:25:03.483404 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.483363 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510"} err="failed to get container status \"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510\": rpc error: code = NotFound desc = could not find container \"46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510\": container with ID starting with 46dcdc8db247dbe9ffb972ce2155906c288495da538656f888f1c43239c0b510 not found: ID does not exist" Jul 18 12:25:03.494394 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.494370 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:25:03.499643 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.499605 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07bfbcac-5dbdd5cc7-c6kg4"] Jul 18 12:25:03.557350 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.557320 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0444c3d5-a18a-4c5f-90c0-30eebb217127-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:25:03.557350 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.557343 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxfc\" (UniqueName: \"kubernetes.io/projected/0444c3d5-a18a-4c5f-90c0-30eebb217127-kube-api-access-njxfc\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:25:03.955476 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:03.955444 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" path="/var/lib/kubelet/pods/0444c3d5-a18a-4c5f-90c0-30eebb217127/volumes" Jul 18 12:25:07.743162 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:07.743130 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:08.109491 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:08.109459 2573 scope.go:117] "RemoveContainer" containerID="31c2bdefcf67f1441b9406f9cb1f48feba00ac9be32289af00a0b1c15ee4b1ff" Jul 18 12:25:09.862909 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:09.862870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:10.464676 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:10.464641 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:25:14.884202 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:14.884162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:51.277902 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:51.277841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:51.791849 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:51.791817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:57.418426 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:57.418362 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:57.489009 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:57.488964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:57.742865 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:57.742769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:57.963517 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:57.963484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:58.977420 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:58.977385 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:59.082534 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:59.082497 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:59.176511 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:59.176477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:59.878453 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:59.878410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:25:59.977346 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:25:59.977305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:00.077347 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:00.077316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:01.370787 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:01.370758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:01.769738 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:01.769691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:26:07.259928 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:07.259895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:07.320588 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:07.320551 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:07.913252 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:07.913208 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:08.133586 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:08.133553 2573 scope.go:117] "RemoveContainer" containerID="8a1c67ff1eedbe1a81063e14d57801fd6a1249db05eff211a9f569369bc88595" Jul 18 12:26:13.734739 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:13.734674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:13.838687 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:13.838641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:13.936194 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:13.936155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:13.975373 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:13.975339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:26:24.418245 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:24.418205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:24.749838 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:24.749798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:45.703559 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:45.703525 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:51.734295 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:51.734261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:51.835478 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:51.835437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:52.065935 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:52.065898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:52.418587 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:52.418494 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:52.597555 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:52.597516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:55.756597 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.756560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:55.848006 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.847975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:55.852174 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.852151 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:26:55.852572 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.852555 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerName="maas-api" Jul 18 12:26:55.852654 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.852575 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerName="maas-api" Jul 18 12:26:55.852738 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.852664 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0444c3d5-a18a-4c5f-90c0-30eebb217127" containerName="maas-api" Jul 18 12:26:55.855666 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.855641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:55.857949 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.857928 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3be5a3-serving-cert\"" Jul 18 12:26:55.868019 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.867993 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:26:55.926335 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.926302 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:55.947811 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.947772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:55.947997 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.947873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:55.947997 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:55.947916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf5j\" (UniqueName: \"kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.023500 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.023408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:56.050409 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.050365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.050591 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.050461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.050591 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.050505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf5j\" (UniqueName: \"kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.051205 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.051183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.053255 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.053233 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.062358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.062338 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf5j\" (UniqueName: \"kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j\") pod \"maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.165888 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.165853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:56.302825 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.302745 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:26:56.305209 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:26:56.305176 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59df34d_9e2c_4716_b2ee_9aba0bebeb77.slice/crio-923bcd6244109a6dd4e37ab66af454126ca907fb1f2aeba1a86b608765809b7d WatchSource:0}: Error finding container 923bcd6244109a6dd4e37ab66af454126ca907fb1f2aeba1a86b608765809b7d: Status 404 returned error can't find the container with id 923bcd6244109a6dd4e37ab66af454126ca907fb1f2aeba1a86b608765809b7d Jul 18 12:26:56.426459 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.426413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:56.525489 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.525462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:26:56.996931 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.996608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" event={"ID":"a59df34d-9e2c-4716-b2ee-9aba0bebeb77","Type":"ContainerStarted","Data":"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f"} Jul 18 12:26:56.996931 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.996675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" event={"ID":"a59df34d-9e2c-4716-b2ee-9aba0bebeb77","Type":"ContainerStarted","Data":"923bcd6244109a6dd4e37ab66af454126ca907fb1f2aeba1a86b608765809b7d"} Jul 18 12:26:56.997386 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:56.996953 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:26:57.015046 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:26:57.015001 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" podStartSLOduration=1.5777475669999999 podStartE2EDuration="2.014984396s" totalImagesPullingTime="437.236829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:26:57.013575376 +0000 UTC m=+2269.504347523" watchObservedRunningTime="2026-07-18 12:26:57.014984396 +0000 UTC m=+2269.505756531" Jul 18 12:27:00.250253 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:00.250217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:00.567284 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:00.567203 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:00.864905 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:00.864815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:01.269275 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:01.269241 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:27:01.724664 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:01.724570 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:01.825179 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:01.825145 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:03.007314 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.007288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:27:03.757264 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.757211 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:03.851764 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.851734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:03.947504 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.947474 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:27:03.951312 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.951280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:03.953778 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.953753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f133b3-serving-cert\"" Jul 18 12:27:03.964075 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.963198 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:27:03.990109 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:03.990074 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:04.017114 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.017015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4r8\" (UniqueName: \"kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.017498 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.017139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.017498 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.017176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.118141 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.118097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4r8\" (UniqueName: \"kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.118362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.118211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.118362 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.118233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.118496 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:27:04.118380 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-serving-cert: secret "maas-api-e2e-inf-b-f133b3-serving-cert" not found Jul 18 12:27:04.118496 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:27:04.118491 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls podName:c8abe0e0-1181-4672-9b62-45fb484ea0ec nodeName:}" failed. No retries permitted until 2026-07-18 12:27:04.618468977 +0000 UTC m=+2277.109241087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls") pod "maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" (UID: "c8abe0e0-1181-4672-9b62-45fb484ea0ec") : secret "maas-api-e2e-inf-b-f133b3-serving-cert" not found Jul 18 12:27:04.118903 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.118885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.128167 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.128132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4r8\" (UniqueName: \"kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.223377 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.223335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:04.624068 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.624036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.626625 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.626594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:04.699566 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.699531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:04.800416 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.800383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:04.865809 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:04.865778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:05.006282 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:05.006248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:27:05.014538 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:27:05.014513 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8abe0e0_1181_4672_9b62_45fb484ea0ec.slice/crio-8f3628f80cbbf223b6c0380a2da70cf2abfcb9542c4dca87cecadebd87a3069f WatchSource:0}: Error finding container 8f3628f80cbbf223b6c0380a2da70cf2abfcb9542c4dca87cecadebd87a3069f: Status 404 returned error can't find the container with id 8f3628f80cbbf223b6c0380a2da70cf2abfcb9542c4dca87cecadebd87a3069f Jul 18 12:27:05.036564 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:05.036533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" event={"ID":"c8abe0e0-1181-4672-9b62-45fb484ea0ec","Type":"ContainerStarted","Data":"8f3628f80cbbf223b6c0380a2da70cf2abfcb9542c4dca87cecadebd87a3069f"} Jul 18 12:27:06.043277 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:06.043243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" event={"ID":"c8abe0e0-1181-4672-9b62-45fb484ea0ec","Type":"ContainerStarted","Data":"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441"} Jul 18 12:27:06.043690 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:06.043341 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:06.062570 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:06.062519 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" podStartSLOduration=2.428085591 podStartE2EDuration="3.062503076s" totalImagesPullingTime="634.417485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:27:06.0616146 +0000 UTC m=+2278.552386720" watchObservedRunningTime="2026-07-18 12:27:06.062503076 +0000 UTC m=+2278.553275192" Jul 18 12:27:08.152429 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:08.152396 2573 scope.go:117] "RemoveContainer" containerID="d5e0e026a6f1785afc548f6e6bbede722e6cb0fd0bf230dfcf821fcfb0b77176" Jul 18 12:27:08.217764 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:08.217734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:09.769657 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:09.769619 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:27:12.053152 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:12.053122 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:27:17.261761 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:17.261696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:17.362273 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:17.362233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:23.149186 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:23.149147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:23.351799 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:23.351761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:23.792880 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:23.792849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:25.490394 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:25.490347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:25.590951 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:25.590912 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:28.173375 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:28.173334 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:27:48.467460 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:48.467389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:27:53.685493 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:27:53.685461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:28:20.939661 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:20.939618 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:22.278140 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:22.278104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:23.219244 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:23.219209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:23.981739 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:23.981681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:24.070308 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:24.070262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:27.165219 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:27.165181 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:28:33.443358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:33.443316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:33.946062 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:33.946019 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:28:33.946294 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:33.946260 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" podUID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" containerName="maas-api" containerID="cri-o://11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f" gracePeriod=30 Jul 18 12:28:34.202014 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.201950 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:28:34.333182 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.333136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca\") pod \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " Jul 18 12:28:34.333182 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.333189 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls\") pod \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " Jul 18 12:28:34.333427 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.333237 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzf5j\" (UniqueName: \"kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j\") pod \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\" (UID: \"a59df34d-9e2c-4716-b2ee-9aba0bebeb77\") " Jul 18 12:28:34.333536 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.333515 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a59df34d-9e2c-4716-b2ee-9aba0bebeb77" (UID: "a59df34d-9e2c-4716-b2ee-9aba0bebeb77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:28:34.335495 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.335464 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j" (OuterVolumeSpecName: "kube-api-access-rzf5j") pod "a59df34d-9e2c-4716-b2ee-9aba0bebeb77" (UID: "a59df34d-9e2c-4716-b2ee-9aba0bebeb77"). InnerVolumeSpecName "kube-api-access-rzf5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:34.335636 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.335572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a59df34d-9e2c-4716-b2ee-9aba0bebeb77" (UID: "a59df34d-9e2c-4716-b2ee-9aba0bebeb77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:28:34.434773 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.434737 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:34.434773 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.434768 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:34.434773 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.434778 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf5j\" (UniqueName: \"kubernetes.io/projected/a59df34d-9e2c-4716-b2ee-9aba0bebeb77-kube-api-access-rzf5j\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:34.455784 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.455678 2573 generic.go:356] "Generic (PLEG): container finished" podID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" containerID="11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f" exitCode=0 Jul 18 12:28:34.455784 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.455736 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" Jul 18 12:28:34.455784 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.455765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" event={"ID":"a59df34d-9e2c-4716-b2ee-9aba0bebeb77","Type":"ContainerDied","Data":"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f"} Jul 18 12:28:34.456265 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.455817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr" event={"ID":"a59df34d-9e2c-4716-b2ee-9aba0bebeb77","Type":"ContainerDied","Data":"923bcd6244109a6dd4e37ab66af454126ca907fb1f2aeba1a86b608765809b7d"} Jul 18 12:28:34.456265 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.455838 2573 scope.go:117] "RemoveContainer" containerID="11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f" Jul 18 12:28:34.465320 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.465301 2573 scope.go:117] "RemoveContainer" containerID="11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f" Jul 18 12:28:34.465582 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:28:34.465564 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f\": container with ID starting with 11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f not found: ID does not exist" containerID="11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f" Jul 18 12:28:34.465663 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.465593 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f"} err="failed to get container status \"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f\": rpc error: code = NotFound desc = could not find container \"11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f\": container with ID starting with 11f4e45f278bdfac30a489be239a1c9b0d3109f7a6517b724079c6dfd031ec7f not found: ID does not exist" Jul 18 12:28:34.491509 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.491470 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:28:34.503965 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.503938 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3be5a3-86499f9478-c26lr"] Jul 18 12:28:34.568584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:34.568529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:35.954600 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:35.954565 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" path="/var/lib/kubelet/pods/a59df34d-9e2c-4716-b2ee-9aba0bebeb77/volumes" Jul 18 12:28:38.413462 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:38.413410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:38.875416 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:38.875377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:43.286120 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:43.286082 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:28:45.386990 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:45.386956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:45.485767 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:45.485733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:45.727927 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:45.727834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:45.951726 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:45.951667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:46.097026 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.096995 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l"] Jul 18 12:28:46.097370 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.097357 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" containerName="maas-api" Jul 18 12:28:46.097370 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.097370 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" containerName="maas-api" Jul 18 12:28:46.097501 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.097435 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59df34d-9e2c-4716-b2ee-9aba0bebeb77" containerName="maas-api" Jul 18 12:28:46.100651 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.100629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:46.107984 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.107960 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:28:46.115313 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.115287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l"] Jul 18 12:28:46.123503 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.123481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l\" (UID: \"e71ef580-7780-45d9-90cb-5e231afbda63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:46.224180 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.224147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l\" (UID: \"e71ef580-7780-45d9-90cb-5e231afbda63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:46.224753 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.224736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l\" (UID: \"e71ef580-7780-45d9-90cb-5e231afbda63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:46.414410 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.414319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:46.591620 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.591592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l"] Jul 18 12:28:46.593779 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:28:46.593752 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ef580_7780_45d9_90cb_5e231afbda63.slice/crio-ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd WatchSource:0}: Error finding container ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd: Status 404 returned error can't find the container with id ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd Jul 18 12:28:46.952566 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:46.952537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:47.059010 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:47.058978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:47.518524 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:47.518488 2573 generic.go:356] "Generic (PLEG): container finished" podID="e71ef580-7780-45d9-90cb-5e231afbda63" containerID="6502b04a9272755fe3e3c43769e1f414ee625e76c6d156974f9a3a7e890eb595" exitCode=0 Jul 18 12:28:47.518943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:47.518525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" event={"ID":"e71ef580-7780-45d9-90cb-5e231afbda63","Type":"ContainerDied","Data":"6502b04a9272755fe3e3c43769e1f414ee625e76c6d156974f9a3a7e890eb595"} Jul 18 12:28:47.518943 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:47.518590 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" event={"ID":"e71ef580-7780-45d9-90cb-5e231afbda63","Type":"ContainerStarted","Data":"ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd"} Jul 18 12:28:48.594056 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.594030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:48.663152 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.663127 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:48.702359 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.702331 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:48.743534 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.743503 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca\") pod \"e71ef580-7780-45d9-90cb-5e231afbda63\" (UID: \"e71ef580-7780-45d9-90cb-5e231afbda63\") " Jul 18 12:28:48.743902 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.743883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e71ef580-7780-45d9-90cb-5e231afbda63" (UID: "e71ef580-7780-45d9-90cb-5e231afbda63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:28:48.844201 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:48.844122 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ef580-7780-45d9-90cb-5e231afbda63-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:49.533538 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:49.533508 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" Jul 18 12:28:49.533731 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:49.533534 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l" event={"ID":"e71ef580-7780-45d9-90cb-5e231afbda63","Type":"ContainerDied","Data":"ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd"} Jul 18 12:28:49.533731 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:49.533598 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4f2a124dc7ed5e51f97f1335c6b5ae9dd62a65dc4389122e549ea51a9dd2bd" Jul 18 12:28:56.925203 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:56.925167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:57.024533 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.024500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:57.201476 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.201390 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:28:57.201952 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.201900 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" podUID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" containerName="maas-api" containerID="cri-o://838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441" gracePeriod=30 Jul 18 12:28:57.254038 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.254011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:28:57.452054 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.451987 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:28:57.522622 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.522592 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") pod \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " Jul 18 12:28:57.522815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.522648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx4r8\" (UniqueName: \"kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8\") pod \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " Jul 18 12:28:57.522815 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.522679 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca\") pod \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\" (UID: \"c8abe0e0-1181-4672-9b62-45fb484ea0ec\") " Jul 18 12:28:57.523121 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.523096 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8abe0e0-1181-4672-9b62-45fb484ea0ec" (UID: "c8abe0e0-1181-4672-9b62-45fb484ea0ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:28:57.524851 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.524831 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8abe0e0-1181-4672-9b62-45fb484ea0ec" (UID: "c8abe0e0-1181-4672-9b62-45fb484ea0ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:28:57.524974 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.524950 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8" (OuterVolumeSpecName: "kube-api-access-gx4r8") pod "c8abe0e0-1181-4672-9b62-45fb484ea0ec" (UID: "c8abe0e0-1181-4672-9b62-45fb484ea0ec"). InnerVolumeSpecName "kube-api-access-gx4r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:57.569882 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.569847 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" containerID="838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441" exitCode=0 Jul 18 12:28:57.570025 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.569888 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" Jul 18 12:28:57.570025 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.569939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" event={"ID":"c8abe0e0-1181-4672-9b62-45fb484ea0ec","Type":"ContainerDied","Data":"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441"} Jul 18 12:28:57.570025 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.569998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w" event={"ID":"c8abe0e0-1181-4672-9b62-45fb484ea0ec","Type":"ContainerDied","Data":"8f3628f80cbbf223b6c0380a2da70cf2abfcb9542c4dca87cecadebd87a3069f"} Jul 18 12:28:57.570210 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.570026 2573 scope.go:117] "RemoveContainer" containerID="838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441" Jul 18 12:28:57.578640 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.578620 2573 scope.go:117] "RemoveContainer" containerID="838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441" Jul 18 12:28:57.578912 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:28:57.578896 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441\": container with ID starting with 838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441 not found: ID does not exist" containerID="838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441" Jul 18 12:28:57.578968 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.578922 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441"} err="failed to get container status \"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441\": rpc error: code = NotFound desc = could not find container \"838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441\": container with ID starting with 838f1ab5bbd5569ee52eac10919880b57560e5f7257208b4bef48c5e1fb97441 not found: ID does not exist" Jul 18 12:28:57.591608 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.591585 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:28:57.595547 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.595522 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f133b3-75b7767f66-zsp2w"] Jul 18 12:28:57.624134 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.624114 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8abe0e0-1181-4672-9b62-45fb484ea0ec-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:57.624239 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.624137 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4r8\" (UniqueName: \"kubernetes.io/projected/c8abe0e0-1181-4672-9b62-45fb484ea0ec-kube-api-access-gx4r8\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:57.624239 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.624150 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8abe0e0-1181-4672-9b62-45fb484ea0ec-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:28:57.954668 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:57.954634 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" path="/var/lib/kubelet/pods/c8abe0e0-1181-4672-9b62-45fb484ea0ec/volumes" Jul 18 12:28:59.901541 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:28:59.901505 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:29:01.223031 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:01.222985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:01.231353 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:01.231328 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l"] Jul 18 12:29:01.237557 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:01.237535 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f133b3-xks8l"] Jul 18 12:29:01.635670 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:01.635638 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:01.954624 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:01.954546 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71ef580-7780-45d9-90cb-5e231afbda63" path="/var/lib/kubelet/pods/e71ef580-7780-45d9-90cb-5e231afbda63/volumes" Jul 18 12:29:03.147333 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:03.147293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:07.801023 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:07.800979 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:07.902951 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:07.902919 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:08.039038 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:08.039004 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:29:08.056258 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:08.056171 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:29:08.114359 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:08.114320 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:08.355531 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:08.355417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:08.685170 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:08.685075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:09.011286 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:09.011241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:09.212770 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:09.212734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:11.522926 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.522890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:11.621561 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.621521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:11.622897 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.622877 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:11.623229 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623217 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" containerName="maas-api" Jul 18 12:29:11.623272 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623231 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" containerName="maas-api" Jul 18 12:29:11.623272 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623240 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ef580-7780-45d9-90cb-5e231afbda63" containerName="revoke-keys" Jul 18 12:29:11.623272 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623246 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ef580-7780-45d9-90cb-5e231afbda63" containerName="revoke-keys" Jul 18 12:29:11.623364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623300 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71ef580-7780-45d9-90cb-5e231afbda63" containerName="revoke-keys" Jul 18 12:29:11.623364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.623308 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8abe0e0-1181-4672-9b62-45fb484ea0ec" containerName="maas-api" Jul 18 12:29:11.627335 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.627317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.631237 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.631116 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-07813a-serving-cert\"" Jul 18 12:29:11.639279 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.639256 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:11.709125 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.709094 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:11.761183 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.761147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.761361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.761192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs4s\" (UniqueName: \"kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.761361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.761232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.862218 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.862131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.862369 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.862255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.862369 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.862296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs4s\" (UniqueName: \"kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.862439 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:11.862408 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-serving-cert: secret "maas-api-e2e-api-a-07813a-serving-cert" not found Jul 18 12:29:11.862497 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:11.862486 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls podName:e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f nodeName:}" failed. No retries permitted until 2026-07-18 12:29:12.362468155 +0000 UTC m=+2404.853240265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls") pod "maas-api-e2e-api-a-07813a-d47c579db-9j7q4" (UID: "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f") : secret "maas-api-e2e-api-a-07813a-serving-cert" not found Jul 18 12:29:11.862891 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.862873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:11.872591 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:11.872567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs4s\" (UniqueName: \"kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:12.218202 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:12.218106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:12.333298 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:12.333262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:12.367484 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:12.367454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:12.367682 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:12.367627 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-serving-cert: secret "maas-api-e2e-api-a-07813a-serving-cert" not found Jul 18 12:29:12.367784 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:12.367733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls podName:e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f nodeName:}" failed. No retries permitted until 2026-07-18 12:29:13.367695442 +0000 UTC m=+2405.858467540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls") pod "maas-api-e2e-api-a-07813a-d47c579db-9j7q4" (UID: "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f") : secret "maas-api-e2e-api-a-07813a-serving-cert" not found Jul 18 12:29:12.603739 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:12.603685 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:13.377914 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.377883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:13.380384 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.380352 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"maas-api-e2e-api-a-07813a-d47c579db-9j7q4\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:13.438817 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.438780 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:13.586784 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.586760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:13.589396 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:29:13.589369 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd9cae_2bd4_4c75_9e00_823f29eb0f9f.slice/crio-82f600a0790d3396d7b93bf32130111fd84ee6dfa12ea92f6a9766125e76b96a WatchSource:0}: Error finding container 82f600a0790d3396d7b93bf32130111fd84ee6dfa12ea92f6a9766125e76b96a: Status 404 returned error can't find the container with id 82f600a0790d3396d7b93bf32130111fd84ee6dfa12ea92f6a9766125e76b96a Jul 18 12:29:13.590822 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.590805 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 12:29:13.645441 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:13.645359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" event={"ID":"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f","Type":"ContainerStarted","Data":"82f600a0790d3396d7b93bf32130111fd84ee6dfa12ea92f6a9766125e76b96a"} Jul 18 12:29:14.652010 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:14.651962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" event={"ID":"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f","Type":"ContainerStarted","Data":"0285f73c881c9caf65bccf87dc385f958f7455438395ae71b279a7014fe0f65d"} Jul 18 12:29:14.652391 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:14.652059 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:14.674125 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:14.674074 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" podStartSLOduration=3.321445304 podStartE2EDuration="3.674057659s" totalImagesPullingTime="352.612355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:29:14.671290043 +0000 UTC m=+2407.162062160" watchObservedRunningTime="2026-07-18 12:29:14.674057659 +0000 UTC m=+2407.164829776" Jul 18 12:29:15.972967 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:15.972935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:16.473966 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:16.473929 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:16.793381 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:16.793341 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:16.883733 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:16.883674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:18.943161 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:18.943133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:29:19.728169 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:19.728135 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:19.912906 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:19.912873 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:29:19.916398 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:19.916383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:19.919472 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:19.919443 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b7127d-serving-cert\"" Jul 18 12:29:19.958431 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:19.958396 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:29:20.037993 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.037959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gp44\" (UniqueName: \"kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.038191 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.038015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.038191 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.038059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.041493 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.041457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:20.139222 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.139179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.139222 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.139219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.139492 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.139286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gp44\" (UniqueName: \"kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.139492 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:20.139388 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-serving-cert: secret "maas-api-e2e-api-b-b7127d-serving-cert" not found Jul 18 12:29:20.139492 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:29:20.139469 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls podName:420020a9-ceb6-4a77-885a-e21a107f489c nodeName:}" failed. No retries permitted until 2026-07-18 12:29:20.639447138 +0000 UTC m=+2413.130219244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls") pod "maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" (UID: "420020a9-ceb6-4a77-885a-e21a107f489c") : secret "maas-api-e2e-api-b-b7127d-serving-cert" not found Jul 18 12:29:20.139880 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.139862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.159481 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.159446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gp44\" (UniqueName: \"kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.176855 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.176825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:20.327121 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.327027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:20.643740 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.643631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.646295 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.646266 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") pod \"maas-api-e2e-api-b-b7127d-b5b764784-kh6z8\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.662145 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.662121 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:20.826086 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.826042 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:20.975821 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.975786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:20.984961 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:20.984940 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:29:20.987662 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:29:20.987631 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420020a9_ceb6_4a77_885a_e21a107f489c.slice/crio-1765cfaadae4868900040c0d7dc59399ae10fc4429a54bd59eb7e22bb9554407 WatchSource:0}: Error finding container 1765cfaadae4868900040c0d7dc59399ae10fc4429a54bd59eb7e22bb9554407: Status 404 returned error can't find the container with id 1765cfaadae4868900040c0d7dc59399ae10fc4429a54bd59eb7e22bb9554407 Jul 18 12:29:21.171738 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:21.171614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:21.687465 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:21.687425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" event={"ID":"420020a9-ceb6-4a77-885a-e21a107f489c","Type":"ContainerStarted","Data":"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6"} Jul 18 12:29:21.687659 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:21.687488 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" event={"ID":"420020a9-ceb6-4a77-885a-e21a107f489c","Type":"ContainerStarted","Data":"1765cfaadae4868900040c0d7dc59399ae10fc4429a54bd59eb7e22bb9554407"} Jul 18 12:29:21.688162 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:21.688134 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:21.718984 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:21.718867 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" podStartSLOduration=2.1899106760000002 podStartE2EDuration="2.718823246s" totalImagesPullingTime="528.91257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:29:21.712102942 +0000 UTC m=+2414.202875053" watchObservedRunningTime="2026-07-18 12:29:21.718823246 +0000 UTC m=+2414.209595371" Jul 18 12:29:24.143160 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:24.143121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:28.702159 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:28.702078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:29:32.054988 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.054949 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x"] Jul 18 12:29:32.058647 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.058628 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:32.061104 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.061083 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:29:32.065896 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.065871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x"] Jul 18 12:29:32.133145 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.133114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07813a-h972x\" (UID: \"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:32.234114 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.234080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07813a-h972x\" (UID: \"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:32.234691 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.234671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07813a-h972x\" (UID: \"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:32.370792 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.370674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:32.506156 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.506126 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x"] Jul 18 12:29:32.507569 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:29:32.507542 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc482384e_5f7a_4c73_8d7c_f2a0d45dbbc9.slice/crio-f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0 WatchSource:0}: Error finding container f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0: Status 404 returned error can't find the container with id f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0 Jul 18 12:29:32.616363 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.616332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:32.715655 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.715611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:32.740330 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:32.740302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" event={"ID":"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9","Type":"ContainerStarted","Data":"f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0"} Jul 18 12:29:33.746820 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:33.746789 2573 generic.go:356] "Generic (PLEG): container finished" podID="c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" containerID="62fa966b0eb565b11a56346ff6431cb9b734dcbceb313a8fdbafdf3c5bf7a6f8" exitCode=0 Jul 18 12:29:33.747214 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:33.746849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" event={"ID":"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9","Type":"ContainerDied","Data":"62fa966b0eb565b11a56346ff6431cb9b734dcbceb313a8fdbafdf3c5bf7a6f8"} Jul 18 12:29:34.692958 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:34.692919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:29:34.894121 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:34.894098 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:34.958004 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:34.957913 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca\") pod \"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9\" (UID: \"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9\") " Jul 18 12:29:34.958291 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:34.958269 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" (UID: "c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:29:35.059072 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:35.059028 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:29:35.757672 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:35.757635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" event={"ID":"c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9","Type":"ContainerDied","Data":"f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0"} Jul 18 12:29:35.757877 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:35.757685 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f563f7684325225c027220740ffc0b2f5e8b5ac14e2fb762858e98cf9ed6d8e0" Jul 18 12:29:35.757877 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:35.757648 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x" Jul 18 12:29:43.385507 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:43.385470 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:43.486291 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:43.486254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:47.615959 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.615928 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:47.616361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.616137 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" podUID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" containerName="maas-api" containerID="cri-o://0285f73c881c9caf65bccf87dc385f958f7455438395ae71b279a7014fe0f65d" gracePeriod=30 Jul 18 12:29:47.638284 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.638252 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:47.815786 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.815751 2573 generic.go:356] "Generic (PLEG): container finished" podID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" containerID="0285f73c881c9caf65bccf87dc385f958f7455438395ae71b279a7014fe0f65d" exitCode=0 Jul 18 12:29:47.815942 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.815829 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" event={"ID":"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f","Type":"ContainerDied","Data":"0285f73c881c9caf65bccf87dc385f958f7455438395ae71b279a7014fe0f65d"} Jul 18 12:29:47.882078 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.882053 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:47.974355 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.974327 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trs4s\" (UniqueName: \"kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s\") pod \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " Jul 18 12:29:47.974531 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.974363 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") pod \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " Jul 18 12:29:47.974531 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.974437 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca\") pod \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\" (UID: \"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f\") " Jul 18 12:29:47.974899 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.974871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" (UID: "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:29:47.976648 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.976614 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" (UID: "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:29:47.976760 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:47.976645 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s" (OuterVolumeSpecName: "kube-api-access-trs4s") pod "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" (UID: "e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f"). InnerVolumeSpecName "kube-api-access-trs4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:29:48.075878 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.075843 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trs4s\" (UniqueName: \"kubernetes.io/projected/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-kube-api-access-trs4s\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:29:48.075878 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.075876 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:29:48.076073 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.075890 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:29:48.821785 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.821748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" event={"ID":"e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f","Type":"ContainerDied","Data":"82f600a0790d3396d7b93bf32130111fd84ee6dfa12ea92f6a9766125e76b96a"} Jul 18 12:29:48.821785 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.821780 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4" Jul 18 12:29:48.822282 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.821826 2573 scope.go:117] "RemoveContainer" containerID="0285f73c881c9caf65bccf87dc385f958f7455438395ae71b279a7014fe0f65d" Jul 18 12:29:48.844107 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.844079 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:48.848443 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:48.848414 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07813a-d47c579db-9j7q4"] Jul 18 12:29:49.955297 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:49.955263 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" path="/var/lib/kubelet/pods/e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f/volumes" Jul 18 12:29:50.478148 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:50.478118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:29:52.161828 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:52.161789 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x"] Jul 18 12:29:52.167302 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:52.167277 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07813a-h972x"] Jul 18 12:29:52.212203 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:52.212172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:52.538563 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:52.538533 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:53.955600 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:53.955567 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" path="/var/lib/kubelet/pods/c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9/volumes" Jul 18 12:29:56.526084 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:56.526048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:58.427968 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:58.427932 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:58.529580 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:58.529551 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:58.688774 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:58.688691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:58.916364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:58.916327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:29:59.038810 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.038781 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4"] Jul 18 12:29:59.039202 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039188 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" containerName="maas-api" Jul 18 12:29:59.039251 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039205 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" containerName="maas-api" Jul 18 12:29:59.039251 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039231 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" containerName="revoke-keys" Jul 18 12:29:59.039251 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039236 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" containerName="revoke-keys" Jul 18 12:29:59.039364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039296 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c482384e-5f7a-4c73-8d7c-f2a0d45dbbc9" containerName="revoke-keys" Jul 18 12:29:59.039364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.039303 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fd9cae-2bd4-4c75-9e00-823f29eb0f9f" containerName="maas-api" Jul 18 12:29:59.043652 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.043625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:29:59.047826 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.047805 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:29:59.054544 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.054518 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4"] Jul 18 12:29:59.173090 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.173050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4\" (UID: \"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:29:59.274316 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.274279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4\" (UID: \"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:29:59.275006 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.274982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4\" (UID: \"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:29:59.354726 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.354633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:29:59.488873 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.488846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4"] Jul 18 12:29:59.491130 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:29:59.491105 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ae1c2c_39b0_4c6f_bba1_656ffca6c321.slice/crio-fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970 WatchSource:0}: Error finding container fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970: Status 404 returned error can't find the container with id fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970 Jul 18 12:29:59.876783 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.876740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" event={"ID":"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321","Type":"ContainerStarted","Data":"b1aba6d5015def0b526ba723d946ac986ef5a8fc516e8036be6bf01e3f9a96ea"} Jul 18 12:29:59.876980 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.876825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" event={"ID":"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321","Type":"ContainerStarted","Data":"fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970"} Jul 18 12:29:59.898130 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:29:59.898073 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" podStartSLOduration=0.898055949 podStartE2EDuration="898.055949ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:29:59.89776316 +0000 UTC m=+2452.388535276" watchObservedRunningTime="2026-07-18 12:29:59.898055949 +0000 UTC m=+2452.388828065" Jul 18 12:30:00.882989 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:00.882953 2573 generic.go:356] "Generic (PLEG): container finished" podID="b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" containerID="b1aba6d5015def0b526ba723d946ac986ef5a8fc516e8036be6bf01e3f9a96ea" exitCode=0 Jul 18 12:30:00.883358 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:00.883037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" event={"ID":"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321","Type":"ContainerDied","Data":"b1aba6d5015def0b526ba723d946ac986ef5a8fc516e8036be6bf01e3f9a96ea"} Jul 18 12:30:02.020601 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.020575 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:30:02.100444 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.100322 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca\") pod \"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321\" (UID: \"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321\") " Jul 18 12:30:02.121066 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.100765 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" (UID: "b1ae1c2c-39b0-4c6f-bba1-656ffca6c321"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:30:02.201720 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.201617 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:30:02.893589 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.893560 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" Jul 18 12:30:02.893589 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.893570 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4" event={"ID":"b1ae1c2c-39b0-4c6f-bba1-656ffca6c321","Type":"ContainerDied","Data":"fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970"} Jul 18 12:30:02.893835 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:02.893612 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe08c6540d6900d48d8e063426fc4c686c806432b7549d3447a282baf4f3a970" Jul 18 12:30:14.534566 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.534526 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:30:14.535163 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.534794 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" podUID="420020a9-ceb6-4a77-885a-e21a107f489c" containerName="maas-api" containerID="cri-o://caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6" gracePeriod=30 Jul 18 12:30:14.561286 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.561246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:14.786767 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.786675 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:30:14.913165 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.913132 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") pod \"420020a9-ceb6-4a77-885a-e21a107f489c\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " Jul 18 12:30:14.913361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.913222 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca\") pod \"420020a9-ceb6-4a77-885a-e21a107f489c\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " Jul 18 12:30:14.913361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.913240 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gp44\" (UniqueName: \"kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44\") pod \"420020a9-ceb6-4a77-885a-e21a107f489c\" (UID: \"420020a9-ceb6-4a77-885a-e21a107f489c\") " Jul 18 12:30:14.913590 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.913565 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "420020a9-ceb6-4a77-885a-e21a107f489c" (UID: "420020a9-ceb6-4a77-885a-e21a107f489c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:30:14.915583 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.915546 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44" (OuterVolumeSpecName: "kube-api-access-6gp44") pod "420020a9-ceb6-4a77-885a-e21a107f489c" (UID: "420020a9-ceb6-4a77-885a-e21a107f489c"). InnerVolumeSpecName "kube-api-access-6gp44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:30:14.915723 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.915594 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "420020a9-ceb6-4a77-885a-e21a107f489c" (UID: "420020a9-ceb6-4a77-885a-e21a107f489c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:30:14.947976 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.947946 2573 generic.go:356] "Generic (PLEG): container finished" podID="420020a9-ceb6-4a77-885a-e21a107f489c" containerID="caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6" exitCode=0 Jul 18 12:30:14.948124 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.947996 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" Jul 18 12:30:14.948124 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.948007 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" event={"ID":"420020a9-ceb6-4a77-885a-e21a107f489c","Type":"ContainerDied","Data":"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6"} Jul 18 12:30:14.948124 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.948049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8" event={"ID":"420020a9-ceb6-4a77-885a-e21a107f489c","Type":"ContainerDied","Data":"1765cfaadae4868900040c0d7dc59399ae10fc4429a54bd59eb7e22bb9554407"} Jul 18 12:30:14.948124 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.948072 2573 scope.go:117] "RemoveContainer" containerID="caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6" Jul 18 12:30:14.956910 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.956892 2573 scope.go:117] "RemoveContainer" containerID="caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6" Jul 18 12:30:14.957186 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:30:14.957163 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6\": container with ID starting with caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6 not found: ID does not exist" containerID="caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6" Jul 18 12:30:14.957252 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.957196 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6"} err="failed to get container status \"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6\": rpc error: code = NotFound desc = could not find container \"caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6\": container with ID starting with caa11db2b470188d45e7fb85192b0b9fd1dc2beacfb5949bdd634e22149de5d6 not found: ID does not exist" Jul 18 12:30:14.970399 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.970371 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:30:14.976885 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:14.976864 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b7127d-b5b764784-kh6z8"] Jul 18 12:30:15.014510 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:15.014463 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420020a9-ceb6-4a77-885a-e21a107f489c-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:30:15.014510 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:15.014504 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gp44\" (UniqueName: \"kubernetes.io/projected/420020a9-ceb6-4a77-885a-e21a107f489c-kube-api-access-6gp44\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:30:15.014510 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:15.014517 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420020a9-ceb6-4a77-885a-e21a107f489c-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:30:15.955832 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:15.955800 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420020a9-ceb6-4a77-885a-e21a107f489c" path="/var/lib/kubelet/pods/420020a9-ceb6-4a77-885a-e21a107f489c/volumes" Jul 18 12:30:19.125731 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.125669 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4"] Jul 18 12:30:19.128587 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.128560 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b7127d-pmbg4"] Jul 18 12:30:19.129121 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.129102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:19.376023 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.375933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:30:19.492221 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.492187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:19.955680 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:19.955633 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" path="/var/lib/kubelet/pods/b1ae1c2c-39b0-4c6f-bba1-656ffca6c321/volumes" Jul 18 12:30:25.441050 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:25.441014 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:25.507324 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:25.507298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:25.716151 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:25.716061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:25.929117 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:25.929072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:26.262938 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:26.262898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:26.571902 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:26.571824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:26.926340 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:26.926247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:29.938184 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:29.938150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:30.037837 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.037808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:30.039595 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.039571 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:30:30.040138 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040122 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420020a9-ceb6-4a77-885a-e21a107f489c" containerName="maas-api" Jul 18 12:30:30.040189 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040143 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="420020a9-ceb6-4a77-885a-e21a107f489c" containerName="maas-api" Jul 18 12:30:30.040189 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040173 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" containerName="revoke-keys" Jul 18 12:30:30.040189 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040183 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" containerName="revoke-keys" Jul 18 12:30:30.040289 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040269 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="420020a9-ceb6-4a77-885a-e21a107f489c" containerName="maas-api" Jul 18 12:30:30.040289 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.040281 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ae1c2c-39b0-4c6f-bba1-656ffca6c321" containerName="revoke-keys" Jul 18 12:30:30.044817 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.044800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.048443 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.048421 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7a27b1-serving-cert\"" Jul 18 12:30:30.055375 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.055342 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:30:30.118849 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.118810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:30.135386 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.135350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprpq\" (UniqueName: \"kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.135541 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.135394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.135541 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.135427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.236810 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.236701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.236985 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.236853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprpq\" (UniqueName: \"kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.236985 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.236887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.237468 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.237450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.239346 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.239326 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.257344 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.257315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprpq\" (UniqueName: \"kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq\") pod \"maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.305847 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.305816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:30.356664 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.356629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:30.490499 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.490420 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:30:30.493850 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:30:30.493822 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f35dfb_9dc4_426e_9635_a218bec698a8.slice/crio-fcfde83afb004d7645213572d0f752ceb56cda3ada1e5544f85b3898de33cff4 WatchSource:0}: Error finding container fcfde83afb004d7645213572d0f752ceb56cda3ada1e5544f85b3898de33cff4: Status 404 returned error can't find the container with id fcfde83afb004d7645213572d0f752ceb56cda3ada1e5544f85b3898de33cff4 Jul 18 12:30:30.650938 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.650900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:30.752689 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:30.752660 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:31.025642 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:31.025614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" event={"ID":"00f35dfb-9dc4-426e-9635-a218bec698a8","Type":"ContainerStarted","Data":"fcfde83afb004d7645213572d0f752ceb56cda3ada1e5544f85b3898de33cff4"} Jul 18 12:30:32.031958 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:32.031925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" event={"ID":"00f35dfb-9dc4-426e-9635-a218bec698a8","Type":"ContainerStarted","Data":"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387"} Jul 18 12:30:32.032346 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:32.032022 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:32.051778 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:32.051729 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" podStartSLOduration=1.5324470799999999 podStartE2EDuration="2.051694088s" totalImagesPullingTime="519.247008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:30:32.049987234 +0000 UTC m=+2484.540759365" watchObservedRunningTime="2026-07-18 12:30:32.051694088 +0000 UTC m=+2484.542466204" Jul 18 12:30:34.403931 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:34.403876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:34.728370 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:34.728290 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:35.012427 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:35.012373 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:35.292169 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:35.292051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:37.169364 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:37.169327 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:30:37.986091 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:37.986054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.043060 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.043035 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:30:38.076275 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.076246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.078915 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.078894 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:30:38.082423 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.082409 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.084618 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.084600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-8b9f97-serving-cert\"" Jul 18 12:30:38.097534 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.097509 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:30:38.102080 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.102055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzttr\" (UniqueName: \"kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.102210 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.102106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.102292 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.102240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.159331 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.159300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.203537 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.203507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.204057 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.203563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzttr\" (UniqueName: \"kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.204057 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.203676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.204057 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:30:38.203814 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-serving-cert: secret "maas-api-e2e-shared-b-8b9f97-serving-cert" not found Jul 18 12:30:38.204057 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:30:38.203897 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls podName:a19b9524-aea5-4492-a2ba-272e45d26adf nodeName:}" failed. No retries permitted until 2026-07-18 12:30:38.703876371 +0000 UTC m=+2491.194648468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls") pod "maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" (UID: "a19b9524-aea5-4492-a2ba-272e45d26adf") : secret "maas-api-e2e-shared-b-8b9f97-serving-cert" not found Jul 18 12:30:38.204355 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.204162 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.215888 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.215864 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzttr\" (UniqueName: \"kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.525426 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.525388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.707875 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.707834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.710404 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.710379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:38.803306 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.803221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.906370 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.906343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:38.998483 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:38.998448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:39.143663 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:39.143595 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:30:39.146006 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:30:39.145980 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19b9524_aea5_4492_a2ba_272e45d26adf.slice/crio-5dcf818f2a518b44da024cc0a7ebb8d3369f67f5d094922cf7e28dbae5783737 WatchSource:0}: Error finding container 5dcf818f2a518b44da024cc0a7ebb8d3369f67f5d094922cf7e28dbae5783737: Status 404 returned error can't find the container with id 5dcf818f2a518b44da024cc0a7ebb8d3369f67f5d094922cf7e28dbae5783737 Jul 18 12:30:40.073753 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:40.073697 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" event={"ID":"a19b9524-aea5-4492-a2ba-272e45d26adf","Type":"ContainerStarted","Data":"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b"} Jul 18 12:30:40.074224 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:40.073776 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:40.074224 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:40.073793 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" event={"ID":"a19b9524-aea5-4492-a2ba-272e45d26adf","Type":"ContainerStarted","Data":"5dcf818f2a518b44da024cc0a7ebb8d3369f67f5d094922cf7e28dbae5783737"} Jul 18 12:30:40.093917 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:40.093870 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" podStartSLOduration=1.712044589 podStartE2EDuration="2.093856039s" totalImagesPullingTime="381.81145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:30:40.091491963 +0000 UTC m=+2492.582264080" watchObservedRunningTime="2026-07-18 12:30:40.093856039 +0000 UTC m=+2492.584628156" Jul 18 12:30:42.436553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:42.436521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:46.085682 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:46.085647 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:30:47.970678 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:47.970631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:30:51.516859 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:51.516820 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:51.617186 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:51.617157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:59.581588 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:59.581455 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:30:59.685520 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:30:59.685483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:31:06.577915 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:31:06.577878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:31:09.766696 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:31:09.766655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:31:26.962619 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:31:26.962577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:31:30.865688 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:31:30.865656 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:31:49.184178 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:31:49.184142 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:32:07.168874 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:32:07.168836 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:32:24.268506 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:32:24.268465 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:32:40.572797 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:32:40.572768 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:32:57.772807 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:32:57.772772 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:33:16.069302 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:16.069266 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:33:28.666013 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:28.665981 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:33:29.543782 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:29.543744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:33:32.282016 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:32.281980 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:33:44.948091 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:44.948051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:33:48.179666 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:48.179622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:33:49.439528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:49.439465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:33:49.545741 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:33:49.545678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:34:08.071430 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:08.071353 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:34:08.091547 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:08.091518 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfgt4_15e43a5d-171f-4982-b708-de1f8ef41752/ovn-acl-logging/0.log" Jul 18 12:34:09.546491 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:09.546452 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:34:15.609310 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:15.609277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:34:19.077097 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:19.077061 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:34:44.464117 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:44.464075 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:34:49.987446 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:49.987410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:34:59.279426 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:59.279388 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:34:59.727302 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:34:59.727205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:04.693596 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:04.693553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:05.476122 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:05.476079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:08.249147 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:08.249118 2573 scope.go:117] "RemoveContainer" containerID="6502b04a9272755fe3e3c43769e1f414ee625e76c6d156974f9a3a7e890eb595" Jul 18 12:35:15.569574 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:15.569541 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:35:27.957893 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:27.957851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:31.168151 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:31.168120 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:35:52.396491 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:52.396453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:52.497209 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:52.497168 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:54.569177 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:54.569142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:35:58.071861 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:35:58.071832 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:36:00.327902 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:00.327861 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:00.430102 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:00.430067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:08.268687 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:08.268653 2573 scope.go:117] "RemoveContainer" containerID="b1aba6d5015def0b526ba723d946ac986ef5a8fc516e8036be6bf01e3f9a96ea" Jul 18 12:36:08.277006 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:08.276982 2573 scope.go:117] "RemoveContainer" containerID="62fa966b0eb565b11a56346ff6431cb9b734dcbceb313a8fdbafdf3c5bf7a6f8" Jul 18 12:36:15.366000 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:15.365965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:36:19.191132 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:19.191095 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:30.165029 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:30.164991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:36:36.755095 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:36.755061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:37.688261 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:37.688219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:46.862874 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:46.862835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:36:57.373376 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:57.373341 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:36:57.478650 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:36:57.478621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:07.171910 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.171354 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:37:07.171910 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.171616 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" podUID="00f35dfb-9dc4-426e-9635-a218bec698a8" containerName="maas-api" containerID="cri-o://462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387" gracePeriod=30 Jul 18 12:37:07.203146 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.203118 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:07.427695 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.427641 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:37:07.550596 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.550559 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hprpq\" (UniqueName: \"kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq\") pod \"00f35dfb-9dc4-426e-9635-a218bec698a8\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " Jul 18 12:37:07.550814 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.550606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca\") pod \"00f35dfb-9dc4-426e-9635-a218bec698a8\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " Jul 18 12:37:07.550814 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.550625 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls\") pod \"00f35dfb-9dc4-426e-9635-a218bec698a8\" (UID: \"00f35dfb-9dc4-426e-9635-a218bec698a8\") " Jul 18 12:37:07.551048 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.551023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00f35dfb-9dc4-426e-9635-a218bec698a8" (UID: "00f35dfb-9dc4-426e-9635-a218bec698a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:37:07.552922 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.552896 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00f35dfb-9dc4-426e-9635-a218bec698a8" (UID: "00f35dfb-9dc4-426e-9635-a218bec698a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:37:07.553007 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.552923 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq" (OuterVolumeSpecName: "kube-api-access-hprpq") pod "00f35dfb-9dc4-426e-9635-a218bec698a8" (UID: "00f35dfb-9dc4-426e-9635-a218bec698a8"). InnerVolumeSpecName "kube-api-access-hprpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:07.652032 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.651995 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hprpq\" (UniqueName: \"kubernetes.io/projected/00f35dfb-9dc4-426e-9635-a218bec698a8-kube-api-access-hprpq\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:07.652032 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.652027 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f35dfb-9dc4-426e-9635-a218bec698a8-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:07.652032 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.652036 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f35dfb-9dc4-426e-9635-a218bec698a8-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:07.910053 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.910019 2573 generic.go:356] "Generic (PLEG): container finished" podID="00f35dfb-9dc4-426e-9635-a218bec698a8" containerID="462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387" exitCode=0 Jul 18 12:37:07.910213 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.910066 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" Jul 18 12:37:07.910213 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.910104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" event={"ID":"00f35dfb-9dc4-426e-9635-a218bec698a8","Type":"ContainerDied","Data":"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387"} Jul 18 12:37:07.910213 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.910154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx" event={"ID":"00f35dfb-9dc4-426e-9635-a218bec698a8","Type":"ContainerDied","Data":"fcfde83afb004d7645213572d0f752ceb56cda3ada1e5544f85b3898de33cff4"} Jul 18 12:37:07.910213 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.910176 2573 scope.go:117] "RemoveContainer" containerID="462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387" Jul 18 12:37:07.919107 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.919092 2573 scope.go:117] "RemoveContainer" containerID="462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387" Jul 18 12:37:07.919345 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:37:07.919325 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387\": container with ID starting with 462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387 not found: ID does not exist" containerID="462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387" Jul 18 12:37:07.919406 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.919356 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387"} err="failed to get container status \"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387\": rpc error: code = NotFound desc = could not find container \"462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387\": container with ID starting with 462e333df8cbcb7f120ef14718b44006acd49f4792762ca481b4f43e52168387 not found: ID does not exist" Jul 18 12:37:07.930631 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.930606 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:37:07.934196 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.934177 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7a27b1-cf66676cf-jnlpx"] Jul 18 12:37:07.955553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:07.955533 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f35dfb-9dc4-426e-9635-a218bec698a8" path="/var/lib/kubelet/pods/00f35dfb-9dc4-426e-9635-a218bec698a8/volumes" Jul 18 12:37:10.070578 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:10.070545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:37:11.840518 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:11.840479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:12.252437 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:12.252406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:18.338033 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.337994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:18.435737 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.435691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:18.597488 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.597396 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:18.827259 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.827199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:18.953779 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.952901 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf"] Jul 18 12:37:18.953779 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.953688 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f35dfb-9dc4-426e-9635-a218bec698a8" containerName="maas-api" Jul 18 12:37:18.957650 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.957520 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f35dfb-9dc4-426e-9635-a218bec698a8" containerName="maas-api" Jul 18 12:37:18.957971 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.957927 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f35dfb-9dc4-426e-9635-a218bec698a8" containerName="maas-api" Jul 18 12:37:18.962701 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.962679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:18.963361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.963336 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf"] Jul 18 12:37:18.965320 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:18.965299 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmjp2\"" Jul 18 12:37:19.051374 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.051346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf\" (UID: \"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:19.151990 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.151955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf\" (UID: \"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:19.152585 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.152565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf\" (UID: \"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:19.275596 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.275570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:19.404840 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.404815 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf"] Jul 18 12:37:19.406922 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:37:19.406894 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c0ce80_0bb3_4bb0_a116_58393d1ba0a3.slice/crio-6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565 WatchSource:0}: Error finding container 6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565: Status 404 returned error can't find the container with id 6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565 Jul 18 12:37:19.488122 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.488095 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:19.587822 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.587703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:19.971404 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.971316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" event={"ID":"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3","Type":"ContainerStarted","Data":"26780465efd51344d09a4b3e7f0b1ccdfd204b224dcc6dabc9e7f3713559a4c8"} Jul 18 12:37:19.971404 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.971374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" event={"ID":"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3","Type":"ContainerStarted","Data":"6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565"} Jul 18 12:37:19.988485 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:19.988443 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" podStartSLOduration=1.988429584 podStartE2EDuration="1.988429584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:37:19.98617217 +0000 UTC m=+2892.476944283" watchObservedRunningTime="2026-07-18 12:37:19.988429584 +0000 UTC m=+2892.479201700" Jul 18 12:37:20.977400 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:20.977366 2573 generic.go:356] "Generic (PLEG): container finished" podID="16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" containerID="26780465efd51344d09a4b3e7f0b1ccdfd204b224dcc6dabc9e7f3713559a4c8" exitCode=0 Jul 18 12:37:20.977801 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:20.977454 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" event={"ID":"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3","Type":"ContainerDied","Data":"26780465efd51344d09a4b3e7f0b1ccdfd204b224dcc6dabc9e7f3713559a4c8"} Jul 18 12:37:22.108312 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.108287 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:22.172285 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.172258 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca\") pod \"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3\" (UID: \"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3\") " Jul 18 12:37:22.172587 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.172566 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" (UID: "16c0ce80-0bb3-4bb0-a116-58393d1ba0a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:37:22.273129 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.273103 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:22.987550 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.987523 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" Jul 18 12:37:22.987736 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.987522 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf" event={"ID":"16c0ce80-0bb3-4bb0-a116-58393d1ba0a3","Type":"ContainerDied","Data":"6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565"} Jul 18 12:37:22.987736 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:22.987650 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af9728a97825841bea1ab92085fc881f8b33d661ea331dc3a681057bbfad565" Jul 18 12:37:29.389222 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.389184 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:37:29.389682 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.389431 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" podUID="a19b9524-aea5-4492-a2ba-272e45d26adf" containerName="maas-api" containerID="cri-o://49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b" gracePeriod=30 Jul 18 12:37:29.422857 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.422832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:29.640572 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.640518 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:37:29.731993 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.731966 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca\") pod \"a19b9524-aea5-4492-a2ba-272e45d26adf\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " Jul 18 12:37:29.732168 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.732017 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzttr\" (UniqueName: \"kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr\") pod \"a19b9524-aea5-4492-a2ba-272e45d26adf\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " Jul 18 12:37:29.732168 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.732064 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") pod \"a19b9524-aea5-4492-a2ba-272e45d26adf\" (UID: \"a19b9524-aea5-4492-a2ba-272e45d26adf\") " Jul 18 12:37:29.732360 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.732336 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a19b9524-aea5-4492-a2ba-272e45d26adf" (UID: "a19b9524-aea5-4492-a2ba-272e45d26adf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:37:29.734070 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.734040 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr" (OuterVolumeSpecName: "kube-api-access-rzttr") pod "a19b9524-aea5-4492-a2ba-272e45d26adf" (UID: "a19b9524-aea5-4492-a2ba-272e45d26adf"). InnerVolumeSpecName "kube-api-access-rzttr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:29.734180 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.734138 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a19b9524-aea5-4492-a2ba-272e45d26adf" (UID: "a19b9524-aea5-4492-a2ba-272e45d26adf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:37:29.833260 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.833222 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19b9524-aea5-4492-a2ba-272e45d26adf-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:29.833260 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.833256 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19b9524-aea5-4492-a2ba-272e45d26adf-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:29.833260 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:29.833265 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzttr\" (UniqueName: \"kubernetes.io/projected/a19b9524-aea5-4492-a2ba-272e45d26adf-kube-api-access-rzttr\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Jul 18 12:37:30.019017 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.018980 2573 generic.go:356] "Generic (PLEG): container finished" podID="a19b9524-aea5-4492-a2ba-272e45d26adf" containerID="49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b" exitCode=0 Jul 18 12:37:30.019153 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.019033 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" Jul 18 12:37:30.019153 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.019064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" event={"ID":"a19b9524-aea5-4492-a2ba-272e45d26adf","Type":"ContainerDied","Data":"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b"} Jul 18 12:37:30.019153 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.019122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t" event={"ID":"a19b9524-aea5-4492-a2ba-272e45d26adf","Type":"ContainerDied","Data":"5dcf818f2a518b44da024cc0a7ebb8d3369f67f5d094922cf7e28dbae5783737"} Jul 18 12:37:30.019153 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.019148 2573 scope.go:117] "RemoveContainer" containerID="49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b" Jul 18 12:37:30.027931 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.027907 2573 scope.go:117] "RemoveContainer" containerID="49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b" Jul 18 12:37:30.028192 ip-10-0-137-172 kubenswrapper[2573]: E0718 12:37:30.028173 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b\": container with ID starting with 49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b not found: ID does not exist" containerID="49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b" Jul 18 12:37:30.028246 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.028202 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b"} err="failed to get container status \"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b\": rpc error: code = NotFound desc = could not find container \"49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b\": container with ID starting with 49eaaa8df015625153619a7072194cd55b5f2c71a92ace667cd8286e1832003b not found: ID does not exist" Jul 18 12:37:30.035673 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.035645 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:37:30.040840 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:30.040819 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8b9f97-666ff65547-l4t8t"] Jul 18 12:37:31.954903 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:31.954875 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19b9524-aea5-4492-a2ba-272e45d26adf" path="/var/lib/kubelet/pods/a19b9524-aea5-4492-a2ba-272e45d26adf/volumes" Jul 18 12:37:32.371045 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:32.371015 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-pmtj9"] Jul 18 12:37:34.025327 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:34.025288 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf"] Jul 18 12:37:34.028526 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:34.028498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:34.028740 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:34.028721 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8b9f97-8wknf"] Jul 18 12:37:34.404564 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:34.404480 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:35.954584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:35.954546 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" path="/var/lib/kubelet/pods/16c0ce80-0bb3-4bb0-a116-58393d1ba0a3/volumes" Jul 18 12:37:40.459300 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:40.459251 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:40.529025 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:40.528986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:40.719744 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:40.719642 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:40.937236 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:40.937202 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:46.535894 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:46.535866 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2dgqx_2f375ae5-c14b-4d08-8641-209764156fac/manager/0.log" Jul 18 12:37:46.653487 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:46.653460 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5v9zn_4bd352bc-fadc-40f7-b772-ed0be93c08db/manager/0.log" Jul 18 12:37:46.775281 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:46.775244 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84ff5bd5f9-vqxz2_07315cc8-1e86-4caa-88fc-d498a3f7f00c/manager/0.log" Jul 18 12:37:46.881670 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:46.881609 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k2wxw_e3204df0-de56-4810-9dbf-43b1c86826bd/server/0.log" Jul 18 12:37:46.999888 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:46.999853 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4ww5d_c4243ffa-19fc-479c-9d31-39a703a4ab0a/manager/0.log" Jul 18 12:37:47.243319 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:47.243247 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-p2ms7_094f6b24-9ca2-4d76-b3d4-50ca9051e262/copy-manifests/0.log" Jul 18 12:37:47.249726 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:47.249692 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-p2ms7_094f6b24-9ca2-4d76-b3d4-50ca9051e262/manager/0.log" Jul 18 12:37:47.702660 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:47.702627 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86bc78d86c-mdn8r_8a59803c-b759-4bdd-92f0-620437320966/maas-api/0.log" Jul 18 12:37:47.808536 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:47.808511 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739600-k2cns_cae6bda8-e5be-4d99-a704-2ecf584f2935/cleanup/0.log" Jul 18 12:37:47.913308 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:47.913278 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739615-f25sq_3cb2aa02-b86a-432e-bf0e-5ed5978db9c2/cleanup/0.log" Jul 18 12:37:48.152464 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:48.152435 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5g5rc_d37511ee-734a-4842-ad9f-74fdaf4153a5/postgres/0.log" Jul 18 12:37:50.068524 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:50.068496 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pmtj9_704f4b8a-95a0-4168-9044-ac0c8450c992/limitador/0.log" Jul 18 12:37:50.655198 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:50.655171 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-flsqs_56e0d20f-a7c3-434f-a3ca-f5989228c73a/discovery/0.log" Jul 18 12:37:50.870286 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:50.870255 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677848b8b7-gfq4w_6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f/kube-auth-proxy/0.log" Jul 18 12:37:51.098306 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:51.098276 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pttt5_8886b023-80b5-46ad-8d30-edbf231cd91b/payload-pre-processing/0.log" Jul 18 12:37:51.642653 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:51.642625 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mntgs_cd7bb9f5-5abe-4a8d-a3dc-9c4e28c51144/main/0.log" Jul 18 12:37:51.962403 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:51.962327 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nmw7_b4995746-508e-4b6c-b0a3-864fc4ccf55e/main/0.log" Jul 18 12:37:57.876724 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:57.876675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:37:57.981688 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:37:57.981655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:38:00.722052 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:00.722018 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xfghz_b665c4bf-22c3-4ea2-8c06-ab541b9d07dc/global-pull-secret-syncer/0.log" Jul 18 12:38:00.778886 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:00.778855 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" Jul 18 12:38:00.902314 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:00.902254 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-172.ec2.internal_bbb9b55189c2ca1c136d2bbd25cb800a/haproxy/0.log" Jul 18 12:38:06.511029 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:06.511002 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-pmtj9_704f4b8a-95a0-4168-9044-ac0c8450c992/limitador/0.log" Jul 18 12:38:08.622695 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.622666 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbwrw_8444571b-e2f0-45d1-a0bf-2d29a060e216/node-exporter/0.log" Jul 18 12:38:08.645457 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.645405 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbwrw_8444571b-e2f0-45d1-a0bf-2d29a060e216/kube-rbac-proxy/0.log" Jul 18 12:38:08.665412 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.665356 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbwrw_8444571b-e2f0-45d1-a0bf-2d29a060e216/init-textfile/0.log" Jul 18 12:38:08.788798 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.788771 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/prometheus/0.log" Jul 18 12:38:08.806584 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.806561 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/config-reloader/0.log" Jul 18 12:38:08.824856 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.824832 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/thanos-sidecar/0.log" Jul 18 12:38:08.843456 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.843437 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/kube-rbac-proxy-web/0.log" Jul 18 12:38:08.865740 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.865700 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/kube-rbac-proxy/0.log" Jul 18 12:38:08.885518 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.885502 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/kube-rbac-proxy-thanos/0.log" Jul 18 12:38:08.905092 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:08.905072 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bf7da949-0300-47cd-af4f-dfe579cfe1f6/init-config-reloader/0.log" Jul 18 12:38:09.902690 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:09.902649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59zmj_6b2d2ab6-93aa-40bb-817e-ef96789eb1b5/konnectivity-agent/0.log" limit=1024 Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.017266 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn"] Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018038 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" containerName="revoke-keys" Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018059 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" containerName="revoke-keys" Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018108 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19b9524-aea5-4492-a2ba-272e45d26adf" containerName="maas-api" Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018118 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19b9524-aea5-4492-a2ba-272e45d26adf" containerName="maas-api" Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018277 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c0ce80-0bb3-4bb0-a116-58393d1ba0a3" containerName="revoke-keys" Jul 18 12:38:11.019937 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.018295 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19b9524-aea5-4492-a2ba-272e45d26adf" containerName="maas-api" Jul 18 12:38:11.023457 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.023429 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.026049 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.026022 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6lsxg\"/\"default-dockercfg-fk6km\"" Jul 18 12:38:11.026167 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.026022 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6lsxg\"/\"openshift-service-ca.crt\"" Jul 18 12:38:11.026658 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.026644 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6lsxg\"/\"kube-root-ca.crt\"" Jul 18 12:38:11.030320 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.030299 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn"] Jul 18 12:38:11.173353 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.173317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-sys\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.173353 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.173348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-podres\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.173553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.173366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-lib-modules\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.173553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.173415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-proc\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.173553 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.173524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpfd\" (UniqueName: \"kubernetes.io/projected/d1c9a06a-8756-4509-ba1b-1a547289140b-kube-api-access-sjpfd\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274291 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpfd\" (UniqueName: \"kubernetes.io/projected/d1c9a06a-8756-4509-ba1b-1a547289140b-kube-api-access-sjpfd\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274291 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-sys\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274291 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-podres\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-lib-modules\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-proc\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-sys\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-proc\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-lib-modules\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.274528 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.274460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1c9a06a-8756-4509-ba1b-1a547289140b-podres\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.282556 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.282527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpfd\" (UniqueName: \"kubernetes.io/projected/d1c9a06a-8756-4509-ba1b-1a547289140b-kube-api-access-sjpfd\") pod \"perf-node-gather-daemonset-hfpjn\" (UID: \"d1c9a06a-8756-4509-ba1b-1a547289140b\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.334361 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.334325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:11.468540 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:11.468513 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn"] Jul 18 12:38:11.470669 ip-10-0-137-172 kubenswrapper[2573]: W0718 12:38:11.470636 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1c9a06a_8756_4509_ba1b_1a547289140b.slice/crio-6c35cdcb3f612bef5ae17c56420f30b5746eeb5f27a4e594e72081957d9162bd WatchSource:0}: Error finding container 6c35cdcb3f612bef5ae17c56420f30b5746eeb5f27a4e594e72081957d9162bd: Status 404 returned error can't find the container with id 6c35cdcb3f612bef5ae17c56420f30b5746eeb5f27a4e594e72081957d9162bd Jul 18 12:38:12.218852 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:12.218815 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" event={"ID":"d1c9a06a-8756-4509-ba1b-1a547289140b","Type":"ContainerStarted","Data":"fd24adc483a594e64ea331a547ec4eef37abd4415809d0ad21b6a622dabff00b"} Jul 18 12:38:12.219238 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:12.218868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" event={"ID":"d1c9a06a-8756-4509-ba1b-1a547289140b","Type":"ContainerStarted","Data":"6c35cdcb3f612bef5ae17c56420f30b5746eeb5f27a4e594e72081957d9162bd"} Jul 18 12:38:12.219238 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:12.218932 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:12.235170 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:12.235125 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" podStartSLOduration=2.235112209 podStartE2EDuration="2.235112209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 12:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:38:12.232939039 +0000 UTC m=+2944.723711156" watchObservedRunningTime="2026-07-18 12:38:12.235112209 +0000 UTC m=+2944.725884325" Jul 18 12:38:13.418899 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:13.418871 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpc88_ce557499-fbb8-43d5-9527-933547932a51/dns/0.log" Jul 18 12:38:13.436568 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:13.436534 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpc88_ce557499-fbb8-43d5-9527-933547932a51/kube-rbac-proxy/0.log" Jul 18 12:38:13.501008 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:13.500989 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t7ntd_d360db2a-a0b4-4f9d-9af7-b34a5a1dff1f/dns-node-resolver/0.log" Jul 18 12:38:14.068502 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:14.068470 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j7d7p_5f205190-9537-4e79-a19b-3b796d3fda6b/node-ca/0.log" Jul 18 12:38:15.245241 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:15.245212 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-flsqs_56e0d20f-a7c3-434f-a3ca-f5989228c73a/discovery/0.log" Jul 18 12:38:15.288079 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:15.288047 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677848b8b7-gfq4w_6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f/kube-auth-proxy/0.log" Jul 18 12:38:15.387372 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:15.387338 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pttt5_8886b023-80b5-46ad-8d30-edbf231cd91b/payload-pre-processing/0.log" Jul 18 12:38:16.125281 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:16.125255 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-445cn_83c93260-093d-4ce7-ab0d-4d238f35cc2b/serve-healthcheck-canary/0.log" Jul 18 12:38:16.649371 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:16.649330 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvxww_e4d73aa4-e12e-4f62-96c8-b91622acbea8/kube-rbac-proxy/0.log" Jul 18 12:38:16.669097 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:16.669069 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvxww_e4d73aa4-e12e-4f62-96c8-b91622acbea8/exporter/0.log" Jul 18 12:38:16.687322 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:16.687291 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvxww_e4d73aa4-e12e-4f62-96c8-b91622acbea8/extractor/0.log" Jul 18 12:38:18.233693 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:18.233669 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-hfpjn" Jul 18 12:38:18.908216 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:18.908190 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2dgqx_2f375ae5-c14b-4d08-8641-209764156fac/manager/0.log" Jul 18 12:38:19.221209 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.221119 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5v9zn_4bd352bc-fadc-40f7-b772-ed0be93c08db/manager/0.log" Jul 18 12:38:19.411768 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.411730 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84ff5bd5f9-vqxz2_07315cc8-1e86-4caa-88fc-d498a3f7f00c/manager/0.log" Jul 18 12:38:19.431438 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.431410 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k2wxw_e3204df0-de56-4810-9dbf-43b1c86826bd/server/0.log" Jul 18 12:38:19.543064 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.543038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4ww5d_c4243ffa-19fc-479c-9d31-39a703a4ab0a/manager/0.log" Jul 18 12:38:19.613772 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.613745 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-p2ms7_094f6b24-9ca2-4d76-b3d4-50ca9051e262/manager/0.log" Jul 18 12:38:19.631523 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:19.631496 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-p2ms7_094f6b24-9ca2-4d76-b3d4-50ca9051e262/copy-manifests/0.log" Jul 18 12:38:21.911500 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:21.911465 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-flsqs_56e0d20f-a7c3-434f-a3ca-f5989228c73a/discovery/0.log" Jul 18 12:38:21.933731 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:21.933686 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677848b8b7-gfq4w_6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f/kube-auth-proxy/0.log" Jul 18 12:38:22.027452 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:22.027366 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pttt5_8886b023-80b5-46ad-8d30-edbf231cd91b/payload-pre-processing/0.log" Jul 18 12:38:23.474524 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:23.474494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-flsqs_56e0d20f-a7c3-434f-a3ca-f5989228c73a/discovery/0.log" Jul 18 12:38:23.504247 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:23.504222 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677848b8b7-gfq4w_6b9a2e6b-cc3e-479d-8e4c-683722a7ad7f/kube-auth-proxy/0.log" Jul 18 12:38:23.588816 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:23.588763 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-pttt5_8886b023-80b5-46ad-8d30-edbf231cd91b/payload-pre-processing/0.log" Jul 18 12:38:26.151463 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:26.151423 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-9gmfq_87b5e01e-3574-4474-9c32-382f6409f5cf/migrator/0.log" Jul 18 12:38:26.168534 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:26.168507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-9gmfq_87b5e01e-3574-4474-9c32-382f6409f5cf/graceful-termination/0.log" Jul 18 12:38:27.555571 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.555538 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5rqww_1866a7a6-57c5-4393-a7bf-6d3cd53e21f1/kube-multus/0.log" Jul 18 12:38:27.881419 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.881343 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/kube-multus-additional-cni-plugins/0.log" Jul 18 12:38:27.898745 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.898723 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/egress-router-binary-copy/0.log" Jul 18 12:38:27.915441 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.915402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/cni-plugins/0.log" Jul 18 12:38:27.933318 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.933296 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/bond-cni-plugin/0.log" Jul 18 12:38:27.952005 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.951986 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/routeoverride-cni/0.log" Jul 18 12:38:27.968909 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.968886 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/whereabouts-cni-bincopy/0.log" Jul 18 12:38:27.985653 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:27.985635 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcss9_afd98e56-e223-4c37-92bb-7a80e4a92e14/whereabouts-cni/0.log" Jul 18 12:38:28.080425 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:28.080401 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrb7g_3b052873-a80a-443c-8a2e-88bca2b4f57f/network-metrics-daemon/0.log" Jul 18 12:38:28.095850 ip-10-0-137-172 kubenswrapper[2573]: I0718 12:38:28.095831 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrb7g_3b052873-a80a-443c-8a2e-88bca2b4f57f/kube-rbac-proxy/0.log"