Jul 16 13:38:20.977072 ip-10-0-130-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 13:38:20.977217 ip-10-0-130-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 13:38:20.977297 ip-10-0-130-14 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 13:38:20.977812 ip-10-0-130-14 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 13:38:31.034667 ip-10-0-130-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 13:38:31.034688 ip-10-0-130-14 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a8906231dbaf47fc97c9b32377a50682 -- Jul 16 13:40:18.904565 ip-10-0-130-14 systemd[1]: Starting Kubernetes Kubelet... Jul 16 13:40:20.285598 ip-10-0-130-14 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 13:40:20.285598 ip-10-0-130-14 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 13:40:20.285598 ip-10-0-130-14 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 13:40:20.285598 ip-10-0-130-14 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 13:40:20.285598 ip-10-0-130-14 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 13:40:20.287143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.287048 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 13:40:20.291255 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291238 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.291255 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291254 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291261 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291266 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291269 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291272 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291275 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291278 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291281 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291284 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291286 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291289 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291292 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291294 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291297 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291301 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291304 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291307 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291310 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291313 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291316 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.291320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291318 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291321 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291324 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291328 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291331 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291334 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291337 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291339 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291342 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291344 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291347 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291349 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291352 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291354 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291357 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291359 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291362 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291365 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291368 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291370 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.291813 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291373 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291376 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291378 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291381 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291383 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291386 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291388 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291391 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291393 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291397 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291399 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291402 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291404 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291408 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291411 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291415 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291417 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291420 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291423 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.292313 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291426 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291429 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291431 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291434 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291436 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291439 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291441 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291444 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291447 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291450 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291453 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291457 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291459 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291462 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291464 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291468 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291473 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291475 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291478 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.292779 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291481 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291483 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291488 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291491 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291493 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291496 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291499 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291502 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291506 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291510 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291512 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291515 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291518 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291521 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291524 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291526 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291529 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291531 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291534 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.291537 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.293242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292263 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292270 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292274 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292278 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292283 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292286 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292289 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292292 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292294 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292297 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292299 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292302 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292305 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292308 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292310 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292313 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292315 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292318 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292321 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.293726 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292324 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292327 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292329 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292332 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292335 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292338 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292340 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292343 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292345 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292350 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292354 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292357 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292360 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292362 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292366 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292369 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292371 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292374 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292376 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292380 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.294229 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292383 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292386 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292388 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292391 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292394 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292396 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292400 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292404 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292407 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292410 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292413 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292417 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292419 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292422 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292425 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292428 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292431 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292433 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292474 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292490 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.294712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292494 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292497 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292501 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292504 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292507 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292510 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292513 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292516 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292519 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292521 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292524 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292532 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292535 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292538 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292541 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292543 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292546 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292549 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292551 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292554 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.295201 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292557 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292559 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292562 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292564 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292567 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292570 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292572 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292575 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292577 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292580 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292582 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292587 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292590 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292593 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292595 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292598 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292601 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292603 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292606 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.292609 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.295670 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293128 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293135 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293139 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293142 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293145 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293148 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293150 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293153 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293156 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293159 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293162 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293165 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293168 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293171 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293173 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293176 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293178 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293181 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293183 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293186 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.296167 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293190 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293193 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293195 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293199 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293202 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293205 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293207 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293210 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293212 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293215 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293217 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293220 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293222 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293225 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293227 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293230 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293232 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293235 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293237 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293240 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.296633 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293243 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293246 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293249 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293252 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293256 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293258 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293261 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293263 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293265 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293268 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293271 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293274 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293276 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293279 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293281 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293284 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293287 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293290 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293292 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293295 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.297119 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293298 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293300 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293303 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293305 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293308 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293310 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293313 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293315 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293318 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293320 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293323 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293325 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293327 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293330 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293333 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293337 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293340 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293342 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293345 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.297608 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293347 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293350 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293352 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293355 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293357 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293361 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293364 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293367 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293373 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293375 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293379 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293381 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293384 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293387 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293390 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293392 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293395 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293398 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293402 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.298077 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.293405 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.293482 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.293489 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294088 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294093 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294098 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294101 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294106 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294111 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294114 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294118 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294121 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294125 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294128 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294131 2563 flags.go:64] FLAG: --cgroup-root="" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294134 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294137 2563 flags.go:64] FLAG: --client-ca-file="" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294140 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294143 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294148 2563 flags.go:64] FLAG: --cluster-domain="" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294151 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294154 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294157 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294161 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 13:40:20.298527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294166 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294169 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294172 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294176 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294179 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294182 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294185 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294188 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294191 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294195 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294197 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294200 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294203 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294206 2563 flags.go:64] FLAG: --enable-server="true" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294210 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294215 2563 flags.go:64] FLAG: --event-burst="100" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294218 2563 flags.go:64] FLAG: --event-qps="50" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294222 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294225 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294228 2563 flags.go:64] FLAG: --eviction-hard="" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294232 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294235 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294239 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294242 2563 flags.go:64] FLAG: --eviction-soft="" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294245 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 13:40:20.299088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294248 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294251 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294254 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294257 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294260 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294264 2563 flags.go:64] FLAG: --feature-gates="" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294267 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294271 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294274 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294278 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294281 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294284 2563 flags.go:64] FLAG: --help="false" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294287 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294290 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294293 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294296 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294299 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294303 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294306 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294309 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294311 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294314 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294317 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294320 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 13:40:20.299712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294323 2563 flags.go:64] FLAG: --kube-reserved="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294326 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294329 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294332 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294335 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294338 2563 flags.go:64] FLAG: --lock-file="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294341 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294344 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294347 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294352 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294355 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294358 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294361 2563 flags.go:64] FLAG: --logging-format="text" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294364 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294368 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294371 2563 flags.go:64] FLAG: --manifest-url="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294373 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294379 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294382 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294386 2563 flags.go:64] FLAG: --max-pods="110" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294389 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294392 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294395 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294398 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294401 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 13:40:20.300383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294404 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294407 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294414 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294417 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294420 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294423 2563 flags.go:64] FLAG: --pod-cidr="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294429 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294436 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294439 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294442 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294445 2563 flags.go:64] FLAG: --port="10250" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294448 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294450 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c79bc5cff5e8f795" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294454 2563 flags.go:64] FLAG: --qos-reserved="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294456 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294459 2563 flags.go:64] FLAG: --register-node="true" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294462 2563 flags.go:64] FLAG: --register-with-taints="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294466 2563 flags.go:64] FLAG: --registry-burst="10" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294469 2563 flags.go:64] FLAG: --registry-qps="5" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294471 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294474 2563 flags.go:64] FLAG: --reserved-memory="" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294478 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294481 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294484 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294487 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 13:40:20.301012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294490 2563 flags.go:64] FLAG: --runonce="false" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294493 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294496 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294499 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294502 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294504 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294507 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294511 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294514 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294516 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294519 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294522 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294525 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294530 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294533 2563 flags.go:64] FLAG: --system-cgroups="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294535 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294541 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294544 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294547 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294550 2563 flags.go:64] FLAG: --tls-min-version="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294553 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294556 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294559 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294562 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294565 2563 flags.go:64] FLAG: --v="2" Jul 16 13:40:20.301606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294569 2563 flags.go:64] FLAG: --version="false" Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294573 2563 flags.go:64] FLAG: --vmodule="" Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294578 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.294582 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294695 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294699 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294702 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294705 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294709 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294714 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294718 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294721 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294723 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294726 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294729 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294731 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294734 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294736 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294739 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294742 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.302212 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294745 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294748 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294751 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294753 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294756 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294758 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294761 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294764 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294767 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294770 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294774 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294777 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294780 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294783 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294786 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294789 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294792 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294794 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294797 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294800 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.302741 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294802 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294807 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294810 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294813 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294816 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294820 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294823 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294826 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294829 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294831 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294834 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294837 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294840 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294843 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294845 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294848 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294851 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294853 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294856 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.303234 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294859 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294861 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294864 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294866 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294869 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294889 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294892 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294896 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294900 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294903 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294906 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294908 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294911 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294914 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294916 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294920 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294922 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294925 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294928 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.303712 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294930 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294933 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294936 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294938 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294941 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294943 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294951 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294953 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294956 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294958 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294961 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294964 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294967 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294969 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294972 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294974 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294977 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294979 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294982 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.304279 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294985 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294988 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294991 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294994 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.294997 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.295000 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.295005 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.301779 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.301797 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.301815 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.301821 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301890 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301896 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301901 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301904 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.304743 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301907 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301910 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301913 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301916 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301918 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301921 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301925 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301927 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301930 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301933 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301935 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301938 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301941 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301944 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301946 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301949 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301952 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301954 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301957 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.305156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301959 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301962 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301964 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301967 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301969 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301972 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301975 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301978 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301981 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301983 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301986 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301989 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301992 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301994 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301997 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.301999 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302002 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302004 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302007 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302009 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.305613 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302012 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302015 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302017 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302020 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302022 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302025 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302029 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302033 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302036 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302038 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302041 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302044 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302046 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302049 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302051 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302054 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302056 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302059 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302062 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302065 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.306182 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302068 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302070 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302073 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302075 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302077 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302080 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302084 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302088 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302090 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302093 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302097 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302100 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302103 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302106 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302109 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302112 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302115 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302118 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302121 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.306671 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302124 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302126 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302129 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302131 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302134 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302136 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302139 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302142 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302144 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302147 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302149 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302152 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302155 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302157 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302160 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302162 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.307224 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302165 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.302170 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302291 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302296 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302299 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302302 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302304 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302307 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302309 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302312 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302315 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302318 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302320 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302323 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302325 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 13:40:20.307984 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302328 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302331 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302333 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302335 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302338 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302340 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302343 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302345 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302348 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302351 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302353 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302356 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302359 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302362 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302365 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302368 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302372 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302375 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302378 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302380 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 13:40:20.308473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302383 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302385 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302388 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302390 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302393 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302395 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302398 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302400 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302403 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302406 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302408 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302411 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302413 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302416 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302418 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302421 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302423 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302426 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302428 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302431 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 13:40:20.309126 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302433 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302436 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302439 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302442 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302445 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302447 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302450 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302453 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302455 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302457 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302460 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302462 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302465 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302467 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302470 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302472 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302475 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302477 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302480 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302482 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 13:40:20.309737 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302484 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302487 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302490 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302493 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302496 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302498 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302501 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302503 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302506 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302508 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302510 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302513 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302515 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302518 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302521 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302524 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302551 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302555 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302558 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 13:40:20.310253 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302561 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302563 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302567 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302569 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302572 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302576 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:20.302579 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.302584 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.302701 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.305768 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.306987 2563 server.go:1019] "Starting client certificate rotation" Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.307109 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.307158 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 13:40:20.310752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.309254 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 13:40:20.312536 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.312505 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 13:40:20.322935 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.322886 2563 log.go:25] "Validated CRI v1 runtime API" Jul 16 13:40:20.326254 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.326235 2563 log.go:25] "Validated CRI v1 image API" Jul 16 13:40:20.327115 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.327101 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 13:40:20.330663 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.330627 2563 fs.go:135] Filesystem UUIDs: map[3ad26a75-4f8b-4aa4-9baa-23953ed2adb8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8fa94e67-6402-4f88-9105-149020b1c50b:/dev/nvme0n1p4] Jul 16 13:40:20.330738 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.330660 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 13:40:20.335661 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.335540 2563 manager.go:216] Machine: {Timestamp:2026-07-16 13:40:20.334336579 +0000 UTC m=+0.282453298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094737 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab2ccaf1748357e0a1c860ec4b879 SystemUUID:ec2ab2cc-af17-4835-7e0a-1c860ec4b879 BootID:a8906231-dbaf-47fc-97c9-b32377a50682 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c0:89:da:ae:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c0:89:da:ae:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:d3:6b:28:52:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 13:40:20.335661 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.335658 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 13:40:20.335762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.335748 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 13:40:20.337949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.337929 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 13:40:20.338834 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.338810 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 13:40:20.338985 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.338837 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 13:40:20.340336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.340324 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 13:40:20.340371 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.340339 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 13:40:20.340371 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.340347 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 13:40:20.340945 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.340934 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 13:40:20.342301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.342289 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 13:40:20.346203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346190 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 16 13:40:20.346323 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346313 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 13:40:20.346363 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346343 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 16 13:40:20.346363 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346361 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 13:40:20.346417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346377 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 13:40:20.346417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346387 2563 kubelet.go:398] "Adding apiserver pod source" Jul 16 13:40:20.346417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346399 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 13:40:20.346864 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346853 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 13:40:20.346913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.346888 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 13:40:20.348779 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.348763 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 13:40:20.350689 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350673 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 13:40:20.350768 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350712 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 13:40:20.350921 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350908 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 13:40:20.350981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350930 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 13:40:20.350981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350939 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 13:40:20.350981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350947 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 13:40:20.350981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350960 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 13:40:20.350981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.350969 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 13:40:20.351370 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351359 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 13:40:20.351423 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351372 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 13:40:20.351423 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351383 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 13:40:20.351423 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351391 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 13:40:20.351423 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351413 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 13:40:20.351550 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.351425 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 13:40:20.352124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.352113 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 13:40:20.352181 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.352127 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 13:40:20.355457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.355439 2563 server.go:1268] "Started kubelet" Jul 16 13:40:20.355603 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.355560 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 13:40:20.355603 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.355535 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 13:40:20.355744 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.355633 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 13:40:20.360038 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.358457 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 13:40:20.358702 ip-10-0-130-14 systemd[1]: Started Kubernetes Kubelet. Jul 16 13:40:20.360630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.360613 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 16 13:40:20.363364 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.363339 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 13:40:20.363470 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.363431 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 13:40:20.363470 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.363447 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 13:40:20.364080 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.364058 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 13:40:20.365651 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.364817 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-14.ec2.internal.18c2c8d53042cd06 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-14.ec2.internal,UID:ip-10-0-130-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-14.ec2.internal,},FirstTimestamp:2026-07-16 13:40:20.355411206 +0000 UTC m=+0.303527904,LastTimestamp:2026-07-16 13:40:20.355411206 +0000 UTC m=+0.303527904,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-14.ec2.internal,}" Jul 16 13:40:20.367682 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.367664 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 13:40:20.367682 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.367670 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 13:40:20.367788 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.367734 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.368249 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368228 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 13:40:20.368374 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368362 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 13:40:20.368486 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368228 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 13:40:20.368587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368493 2563 factory.go:55] Registering systemd factory Jul 16 13:40:20.368587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368509 2563 factory.go:223] Registration of the systemd container factory successfully Jul 16 13:40:20.368587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368550 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 16 13:40:20.368587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368558 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 16 13:40:20.368906 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368889 2563 factory.go:153] Registering CRI-O factory Jul 16 13:40:20.368906 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368905 2563 factory.go:223] Registration of the crio container factory successfully Jul 16 13:40:20.369030 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368956 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 13:40:20.369030 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.368982 2563 factory.go:103] Registering Raw factory Jul 16 13:40:20.369030 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.369000 2563 manager.go:1196] Started watching for new ooms in manager Jul 16 13:40:20.369667 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.369649 2563 manager.go:349] Starting recovery of all containers Jul 16 13:40:20.373039 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.373008 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 13:40:20.373172 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.373134 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 13:40:20.377265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.377124 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 13:40:20.379287 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.379269 2563 manager.go:354] Recovery completed Jul 16 13:40:20.381125 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.381102 2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 16 13:40:20.384776 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.384763 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.386629 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.386606 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.386689 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.386641 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.386689 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.386651 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.387208 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387192 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 13:40:20.387208 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387207 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 13:40:20.387344 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387210 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.387344 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387228 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 16 13:40:20.387887 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387859 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.387948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387903 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.387948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.387914 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.389358 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.389320 2563 policy_none.go:49] "None policy: Start" Jul 16 13:40:20.389358 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.389362 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 13:40:20.389499 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.389376 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 13:40:20.389864 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.389774 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-14.ec2.internal.18c2c8d5321f216c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-14.ec2.internal,UID:ip-10-0-130-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-14.ec2.internal,},FirstTimestamp:2026-07-16 13:40:20.386627948 +0000 UTC m=+0.334744645,LastTimestamp:2026-07-16 13:40:20.386627948 +0000 UTC m=+0.334744645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-14.ec2.internal,}" Jul 16 13:40:20.390597 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.390574 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-574wd" Jul 16 13:40:20.391564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.391549 2563 policy_none.go:47] "Start" Jul 16 13:40:20.396360 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.396344 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-574wd" Jul 16 13:40:20.398308 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.398240 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-14.ec2.internal.18c2c8d5321f68d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-14.ec2.internal,UID:ip-10-0-130-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-14.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-14.ec2.internal,},FirstTimestamp:2026-07-16 13:40:20.386646228 +0000 UTC m=+0.334762926,LastTimestamp:2026-07-16 13:40:20.386646228 +0000 UTC m=+0.334762926,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-14.ec2.internal,}" Jul 16 13:40:20.425600 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.425581 2563 manager.go:337] "Starting Device Plugin manager" Jul 16 13:40:20.425790 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.425617 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 13:40:20.425790 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.425630 2563 server.go:87] "Starting device plugin registration server" Jul 16 13:40:20.425932 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.425920 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 13:40:20.425982 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.425934 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 13:40:20.426060 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.426043 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 13:40:20.426159 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.426139 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 13:40:20.426159 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.426157 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 13:40:20.426729 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.426709 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 13:40:20.426817 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.426756 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.467572 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.467547 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 13:40:20.475421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.467590 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 13:40:20.475421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.467611 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 13:40:20.475421 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.467654 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 13:40:20.475421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.469829 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:20.526947 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.526854 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.528170 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.528152 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.528279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.528190 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.528279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.528208 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.528279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.528240 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.537235 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.537210 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.537235 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.537237 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-14.ec2.internal\": node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.562675 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.562642 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.567761 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.567737 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal"] Jul 16 13:40:20.567845 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.567822 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.569288 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.569272 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.569381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.569301 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.569381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.569314 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.571575 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.571560 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.571748 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.571734 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.571788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.571764 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.572336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572319 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.572417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572347 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.572417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572358 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.572417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572319 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.572417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572386 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.572417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.572399 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.574531 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.574513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.574610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.574542 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 13:40:20.575359 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.575340 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientMemory" Jul 16 13:40:20.575456 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.575373 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 13:40:20.575456 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.575388 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeHasSufficientPID" Jul 16 13:40:20.593530 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.593504 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-14.ec2.internal\" not found" node="ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.597098 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.597079 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-14.ec2.internal\" not found" node="ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.663487 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.663452 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.669306 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.669276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.669442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.669315 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.764528 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.764480 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.769755 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.769739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8976d328a4aa8b80317cf76fe353706f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-14.ec2.internal\" (UID: \"8976d328a4aa8b80317cf76fe353706f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.769812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.769787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.769846 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.769810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.769892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.769851 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.769927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.769892 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291f71208e98001156098a5810a8cb54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal\" (UID: \"291f71208e98001156098a5810a8cb54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.865270 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.865203 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:20.870563 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.870543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8976d328a4aa8b80317cf76fe353706f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-14.ec2.internal\" (UID: \"8976d328a4aa8b80317cf76fe353706f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.870632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.870595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8976d328a4aa8b80317cf76fe353706f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-14.ec2.internal\" (UID: \"8976d328a4aa8b80317cf76fe353706f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.895739 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.895706 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.900833 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:20.900637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:20.965460 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:20.965419 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:21.065957 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.065930 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:21.166610 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.166520 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:21.267057 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.267011 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-14.ec2.internal\" not found" Jul 16 13:40:21.281816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.281787 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:21.285716 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.285697 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:21.306762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.306733 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 13:40:21.306951 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.306928 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 13:40:21.307001 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.306919 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 13:40:21.307001 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.306935 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 13:40:21.347117 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.347083 2563 apiserver.go:52] "Watching apiserver" Jul 16 13:40:21.368016 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.367537 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 13:40:21.368016 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.367760 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 13:40:21.368016 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.367913 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" Jul 16 13:40:21.369257 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.369215 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-kf4n6","openshift-network-diagnostics/network-check-target-g8t8z","kube-system/konnectivity-agent-8f6wm","openshift-cluster-node-tuning-operator/tuned-dvvfh","openshift-dns/node-resolver-lb6pg","openshift-multus/multus-9c256","openshift-multus/multus-additional-cni-plugins-kt66x","openshift-network-operator/iptables-alerter-jn7sh","openshift-ovn-kubernetes/ovnkube-node-ltrtt","kube-system/global-pull-secret-syncer-n9glr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp","openshift-image-registry/node-ca-nqmpw"] Jul 16 13:40:21.371658 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.371641 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.371751 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.371708 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:21.373705 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.373687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:21.373794 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.373737 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:21.375817 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.375802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.377969 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.377935 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 13:40:21.378077 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.378005 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nrmkr\"" Jul 16 13:40:21.378077 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.378062 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.378203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.378186 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 13:40:21.379588 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.379565 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 13:40:21.380045 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.380026 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4gkh\"" Jul 16 13:40:21.380141 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.380094 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.380141 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.380106 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.380277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.380263 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.383621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.382817 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.383621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.383088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.383621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.383364 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rzzm9\"" Jul 16 13:40:21.383621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.383540 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9c256" Jul 16 13:40:21.384966 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.384948 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.385529 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.385506 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 13:40:21.385657 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.385638 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 13:40:21.385826 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.385806 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.385930 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.385862 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.386253 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.386238 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5prbr\"" Jul 16 13:40:21.386894 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.386845 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 13:40:21.387020 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.387003 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rhkqz\"" Jul 16 13:40:21.387109 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.387085 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 13:40:21.387286 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.387271 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.388554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.388534 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 13:40:21.388629 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.388607 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" Jul 16 13:40:21.389223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.389209 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.389360 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.389346 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-66tpr\"" Jul 16 13:40:21.389699 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.389678 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.390057 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.389818 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.390057 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.389829 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 13:40:21.391662 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.391644 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 13:40:21.391812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.391793 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.392019 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392000 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 13:40:21.392293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392274 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.392366 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.392347 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:21.392577 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392561 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bhxvl\"" Jul 16 13:40:21.392627 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392576 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.392808 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392785 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 13:40:21.392905 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.392793 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 13:40:21.394801 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.394784 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.397081 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397050 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 13:40:21.397671 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397652 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.397767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397722 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.397767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397762 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cm7sn\"" Jul 16 13:40:21.397894 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397774 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal"] Jul 16 13:40:21.397954 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.397934 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.398452 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.398434 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 13:40:21.398862 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.398769 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal"] Jul 16 13:40:21.400045 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400027 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 13:40:21.400137 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400082 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 13:35:20 +0000 UTC" deadline="2028-02-19 03:40:25.377816579 +0000 UTC" Jul 16 13:40:21.400193 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400135 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13982h0m3.977685801s" Jul 16 13:40:21.400655 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400612 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 13:40:21.400655 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400639 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpxss\"" Jul 16 13:40:21.400912 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.400870 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 13:40:21.408199 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.408169 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hrht" Jul 16 13:40:21.414391 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.414368 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hrht" Jul 16 13:40:21.470621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.470596 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 13:40:21.473047 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.473128 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473059 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-kubelet-config\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.473128 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cnibin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.473128 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-multus-certs\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.473273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473155 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-conf\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.473273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-etc-selinux\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.473273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473212 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcsf\" (UniqueName: \"kubernetes.io/projected/70c04af3-de33-48d5-9959-2b6c5fd45515-kube-api-access-xkcsf\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.473273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-slash\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2234a173-1018-4893-968c-037f58da8dea-iptables-alerter-script\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-device-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cni-binary-copy\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473345 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473394 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-netd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ef62460-d86f-49bc-9de3-3023d384570f-hosts-file\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.473462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef62460-d86f-49bc-9de3-3023d384570f-tmp-dir\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-os-release\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473507 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-systemd-units\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-konnectivity-ca\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-lib-modules\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-host\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2234a173-1018-4893-968c-037f58da8dea-host-slash\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbwj\" (UniqueName: \"kubernetes.io/projected/cff69847-6d3c-4063-b247-487a7a21d795-kube-api-access-rmbwj\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:21.473757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-var-lib-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473825 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-system-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-bin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-etc-kubernetes\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473892 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-netns\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-etc-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473942 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01a5411a-ae3d-42a2-98cd-926bd4155ac8-serviceca\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-dbus\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.473988 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dht66\" (UniqueName: \"kubernetes.io/projected/2ef62460-d86f-49bc-9de3-3023d384570f-kube-api-access-dht66\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-socket-dir-parent\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvsvc\" (UniqueName: \"kubernetes.io/projected/01a5411a-ae3d-42a2-98cd-926bd4155ac8-kube-api-access-pvsvc\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.474054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-kubernetes\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-tmp\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnmf\" (UniqueName: \"kubernetes.io/projected/8c18e82a-f845-4475-bc78-f29acf434c83-kube-api-access-2wnmf\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-config\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcn5m\" (UniqueName: \"kubernetes.io/projected/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-kube-api-access-pcn5m\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474167 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-sys\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-var-lib-kubelet\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-socket-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-sys-fs\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-multus\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474323 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-node-log\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474372 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-log-socket\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474395 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-tuned\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-k8s-cni-cncf-io\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-conf-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474473 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-ovn\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-env-overrides\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474568 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-script-lib\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a5411a-ae3d-42a2-98cd-926bd4155ac8-host\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474637 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysconfig\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-netns\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-kubelet\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474733 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-hostroot\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474758 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-system-cni-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-bin\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-agent-certs\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-modprobe-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.474847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-systemd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474844 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-run\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpmw\" (UniqueName: \"kubernetes.io/projected/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-kube-api-access-rgpmw\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-kubelet\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovn-node-metrics-cert\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-systemd\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-registration-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-os-release\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.474988 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdsh\" (UniqueName: \"kubernetes.io/projected/8f7287ee-f367-4541-b6b7-08b20a94971d-kube-api-access-bzdsh\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.475002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbmq\" (UniqueName: \"kubernetes.io/projected/2234a173-1018-4893-968c-037f58da8dea-kube-api-access-5qbmq\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.475015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.475041 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-daemon-config\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.475377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.475067 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-cnibin\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.481557 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.481526 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291f71208e98001156098a5810a8cb54.slice/crio-96d4f438b13d33a1ca00912bee8dabbc0453813d1ad965ea38d379c5598bd9ed WatchSource:0}: Error finding container 96d4f438b13d33a1ca00912bee8dabbc0453813d1ad965ea38d379c5598bd9ed: Status 404 returned error can't find the container with id 96d4f438b13d33a1ca00912bee8dabbc0453813d1ad965ea38d379c5598bd9ed Jul 16 13:40:21.486751 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.486733 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:40:21.575862 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-registration-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.575862 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-os-release\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdsh\" (UniqueName: \"kubernetes.io/projected/8f7287ee-f367-4541-b6b7-08b20a94971d-kube-api-access-bzdsh\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-registration-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575970 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-os-release\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.575990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbmq\" (UniqueName: \"kubernetes.io/projected/2234a173-1018-4893-968c-037f58da8dea-kube-api-access-5qbmq\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-daemon-config\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-cnibin\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576075 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-kubelet-config\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cnibin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-cnibin\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576133 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cnibin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-kubelet-config\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-multus-certs\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576184 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-conf\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-multus-certs\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-etc-selinux\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcsf\" (UniqueName: \"kubernetes.io/projected/70c04af3-de33-48d5-9959-2b6c5fd45515-kube-api-access-xkcsf\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576266 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-slash\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2234a173-1018-4893-968c-037f58da8dea-iptables-alerter-script\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576312 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-etc-selinux\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-conf\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576348 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-slash\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-device-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cni-binary-copy\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.576631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-device-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576542 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-netd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ef62460-d86f-49bc-9de3-3023d384570f-hosts-file\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef62460-d86f-49bc-9de3-3023d384570f-tmp-dir\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576648 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-os-release\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576656 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-netd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-systemd-units\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-konnectivity-ca\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-daemon-config\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576755 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ef62460-d86f-49bc-9de3-3023d384570f-hosts-file\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576759 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-systemd-units\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-lib-modules\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-host\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.577346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2234a173-1018-4893-968c-037f58da8dea-host-slash\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576866 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysctl-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbwj\" (UniqueName: \"kubernetes.io/projected/cff69847-6d3c-4063-b247-487a7a21d795-kube-api-access-rmbwj\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576918 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-os-release\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576934 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-host\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2234a173-1018-4893-968c-037f58da8dea-iptables-alerter-script\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576958 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-cni-binary-copy\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576980 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2234a173-1018-4893-968c-037f58da8dea-host-slash\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.576960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-var-lib-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-var-lib-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577055 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577059 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef62460-d86f-49bc-9de3-3023d384570f-tmp-dir\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577088 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-system-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577079 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-lib-modules\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.578150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-bin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577109 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-etc-kubernetes\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-system-cni-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-bin\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-netns\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577231 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-etc-kubernetes\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-etc-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-konnectivity-ca\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577298 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-etc-openvswitch\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01a5411a-ae3d-42a2-98cd-926bd4155ac8-serviceca\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577338 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-dbus\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dht66\" (UniqueName: \"kubernetes.io/projected/2ef62460-d86f-49bc-9de3-3023d384570f-kube-api-access-dht66\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577439 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-socket-dir-parent\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.577464 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577478 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7cc4eb-0314-42c9-ae8d-39152296fbda-dbus\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-run-netns\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.578960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvsvc\" (UniqueName: \"kubernetes.io/projected/01a5411a-ae3d-42a2-98cd-926bd4155ac8-kube-api-access-pvsvc\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.577532 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:22.077515335 +0000 UTC m=+2.025632040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577544 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-socket-dir-parent\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577566 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-kubernetes\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-tmp\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnmf\" (UniqueName: \"kubernetes.io/projected/8c18e82a-f845-4475-bc78-f29acf434c83-kube-api-access-2wnmf\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577619 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-kubernetes\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-config\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcn5m\" (UniqueName: \"kubernetes.io/projected/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-kube-api-access-pcn5m\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577708 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-sys\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01a5411a-ae3d-42a2-98cd-926bd4155ac8-serviceca\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-var-lib-kubelet\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-socket-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577809 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-sys-fs\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577780 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-sys\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577835 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-multus\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577915 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-cni-multus\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.579643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-var-lib-kubelet\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577981 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-socket-dir\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577986 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff69847-6d3c-4063-b247-487a7a21d795-sys-fs\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.577993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-node-log\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578007 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-log-socket\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-tuned\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578052 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-node-log\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-k8s-cni-cncf-io\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-conf-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-log-socket\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-ovn\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578142 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-k8s-cni-cncf-io\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-env-overrides\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578185 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-script-lib\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a5411a-ae3d-42a2-98cd-926bd4155ac8-host\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.580179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a5411a-ae3d-42a2-98cd-926bd4155ac8-host\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-config\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578342 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-ovn\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-multus-conf-dir\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysconfig\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-netns\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-kubelet\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-hostroot\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578560 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-system-cni-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-bin\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-agent-certs\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578811 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-hostroot\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70c04af3-de33-48d5-9959-2b6c5fd45515-system-cni-dir\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578658 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-var-lib-kubelet\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-modprobe-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.580709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-cni-bin\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-sysconfig\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578769 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-host-run-netns\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-systemd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578928 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-run\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-modprobe-d\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpmw\" (UniqueName: \"kubernetes.io/projected/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-kube-api-access-rgpmw\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578996 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-kubelet\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.578736 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovnkube-script-lib\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579040 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovn-node-metrics-cert\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-systemd\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579116 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-run-systemd\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-systemd\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-host-kubelet\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579171 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f7287ee-f367-4541-b6b7-08b20a94971d-run\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70c04af3-de33-48d5-9959-2b6c5fd45515-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.578606 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:21.581334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.579261 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-env-overrides\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.581857 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.579281 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:22.079263315 +0000 UTC m=+2.027380003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:21.581857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.581152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-etc-tuned\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.581223 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f7287ee-f367-4541-b6b7-08b20a94971d-tmp\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.581857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.581322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8489bd65-432f-4e2e-8e58-bd4e2463d7a3-agent-certs\") pod \"konnectivity-agent-8f6wm\" (UID: \"8489bd65-432f-4e2e-8e58-bd4e2463d7a3\") " pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.581857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.581399 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-ovn-node-metrics-cert\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.588138 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.588114 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:21.588138 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.588139 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:21.588319 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.588153 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:21.588319 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:21.588289 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:22.088267698 +0000 UTC m=+2.036384401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:21.588601 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.588575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcsf\" (UniqueName: \"kubernetes.io/projected/70c04af3-de33-48d5-9959-2b6c5fd45515-kube-api-access-xkcsf\") pod \"multus-additional-cni-plugins-kt66x\" (UID: \"70c04af3-de33-48d5-9959-2b6c5fd45515\") " pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.590119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.590089 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbwj\" (UniqueName: \"kubernetes.io/projected/cff69847-6d3c-4063-b247-487a7a21d795-kube-api-access-rmbwj\") pod \"aws-ebs-csi-driver-node-j22rp\" (UID: \"cff69847-6d3c-4063-b247-487a7a21d795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.590289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.590272 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpmw\" (UniqueName: \"kubernetes.io/projected/9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3-kube-api-access-rgpmw\") pod \"multus-9c256\" (UID: \"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3\") " pod="openshift-multus/multus-9c256" Jul 16 13:40:21.590339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.590299 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbmq\" (UniqueName: \"kubernetes.io/projected/2234a173-1018-4893-968c-037f58da8dea-kube-api-access-5qbmq\") pod \"iptables-alerter-jn7sh\" (UID: \"2234a173-1018-4893-968c-037f58da8dea\") " pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.590444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.590422 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnmf\" (UniqueName: \"kubernetes.io/projected/8c18e82a-f845-4475-bc78-f29acf434c83-kube-api-access-2wnmf\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:21.591011 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.590994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcn5m\" (UniqueName: \"kubernetes.io/projected/9e789ac3-b4c4-42d6-85e4-c0dad9024b5e-kube-api-access-pcn5m\") pod \"ovnkube-node-ltrtt\" (UID: \"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.593016 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.592994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdsh\" (UniqueName: \"kubernetes.io/projected/8f7287ee-f367-4541-b6b7-08b20a94971d-kube-api-access-bzdsh\") pod \"tuned-dvvfh\" (UID: \"8f7287ee-f367-4541-b6b7-08b20a94971d\") " pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.593162 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.593145 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvsvc\" (UniqueName: \"kubernetes.io/projected/01a5411a-ae3d-42a2-98cd-926bd4155ac8-kube-api-access-pvsvc\") pod \"node-ca-nqmpw\" (UID: \"01a5411a-ae3d-42a2-98cd-926bd4155ac8\") " pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.593651 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.593628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dht66\" (UniqueName: \"kubernetes.io/projected/2ef62460-d86f-49bc-9de3-3023d384570f-kube-api-access-dht66\") pod \"node-resolver-lb6pg\" (UID: \"2ef62460-d86f-49bc-9de3-3023d384570f\") " pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.694732 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.694643 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:21.704647 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.704618 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8489bd65_432f_4e2e_8e58_bd4e2463d7a3.slice/crio-1813cdb8ab3a319f01dfecfa1254f5350114821fc724da9dd1b62dc09ada2fab WatchSource:0}: Error finding container 1813cdb8ab3a319f01dfecfa1254f5350114821fc724da9dd1b62dc09ada2fab: Status 404 returned error can't find the container with id 1813cdb8ab3a319f01dfecfa1254f5350114821fc724da9dd1b62dc09ada2fab Jul 16 13:40:21.711937 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.711915 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" Jul 16 13:40:21.722361 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.722337 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7287ee_f367_4541_b6b7_08b20a94971d.slice/crio-196932f17c106dfc8a03c7bf7fbf5826536447f22a6a4b13b45c67e285409f05 WatchSource:0}: Error finding container 196932f17c106dfc8a03c7bf7fbf5826536447f22a6a4b13b45c67e285409f05: Status 404 returned error can't find the container with id 196932f17c106dfc8a03c7bf7fbf5826536447f22a6a4b13b45c67e285409f05 Jul 16 13:40:21.724007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.723992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lb6pg" Jul 16 13:40:21.731809 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.731784 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef62460_d86f_49bc_9de3_3023d384570f.slice/crio-c2d4289bb998939def4ecbb19f55b39d42264a80232789538ad2288ef718bf3b WatchSource:0}: Error finding container c2d4289bb998939def4ecbb19f55b39d42264a80232789538ad2288ef718bf3b: Status 404 returned error can't find the container with id c2d4289bb998939def4ecbb19f55b39d42264a80232789538ad2288ef718bf3b Jul 16 13:40:21.743142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.743114 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9c256" Jul 16 13:40:21.751445 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.751418 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa7c6c5_3bea_47c6_81b3_fb8d6f2098d3.slice/crio-9b540b3396b82f867bbfa9922ddf272fa25a27dbbd3a55807ca79e9f82b3107e WatchSource:0}: Error finding container 9b540b3396b82f867bbfa9922ddf272fa25a27dbbd3a55807ca79e9f82b3107e: Status 404 returned error can't find the container with id 9b540b3396b82f867bbfa9922ddf272fa25a27dbbd3a55807ca79e9f82b3107e Jul 16 13:40:21.758453 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.758431 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt66x" Jul 16 13:40:21.764258 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.764242 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jn7sh" Jul 16 13:40:21.767316 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.767221 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c04af3_de33_48d5_9959_2b6c5fd45515.slice/crio-173b1f559dc12c86430e42c9353a5cdff6ea524c6bd43a6106e30254e017873f WatchSource:0}: Error finding container 173b1f559dc12c86430e42c9353a5cdff6ea524c6bd43a6106e30254e017873f: Status 404 returned error can't find the container with id 173b1f559dc12c86430e42c9353a5cdff6ea524c6bd43a6106e30254e017873f Jul 16 13:40:21.773512 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.773486 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2234a173_1018_4893_968c_037f58da8dea.slice/crio-788f3ca9921a299142a80bad43b60df2398041545048721525b6f4fd91b6984f WatchSource:0}: Error finding container 788f3ca9921a299142a80bad43b60df2398041545048721525b6f4fd91b6984f: Status 404 returned error can't find the container with id 788f3ca9921a299142a80bad43b60df2398041545048721525b6f4fd91b6984f Jul 16 13:40:21.784513 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.784493 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:21.790326 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.790303 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" Jul 16 13:40:21.794067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.794044 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqmpw" Jul 16 13:40:21.794136 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.794099 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e789ac3_b4c4_42d6_85e4_c0dad9024b5e.slice/crio-3488ceeb156234f3324302b3af5ecdc8155f7e06665027dab09711232ab892d1 WatchSource:0}: Error finding container 3488ceeb156234f3324302b3af5ecdc8155f7e06665027dab09711232ab892d1: Status 404 returned error can't find the container with id 3488ceeb156234f3324302b3af5ecdc8155f7e06665027dab09711232ab892d1 Jul 16 13:40:21.800552 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.800527 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff69847_6d3c_4063_b247_487a7a21d795.slice/crio-96e7e1a14493dba679979de45cce678568d1efbff2026a72887c6a71317eb347 WatchSource:0}: Error finding container 96e7e1a14493dba679979de45cce678568d1efbff2026a72887c6a71317eb347: Status 404 returned error can't find the container with id 96e7e1a14493dba679979de45cce678568d1efbff2026a72887c6a71317eb347 Jul 16 13:40:21.805324 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:21.805300 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a5411a_ae3d_42a2_98cd_926bd4155ac8.slice/crio-0ae3b2f51b5b1c82263a97d16108ebea631fe33ff5b550bc99349b1fe523a92b WatchSource:0}: Error finding container 0ae3b2f51b5b1c82263a97d16108ebea631fe33ff5b550bc99349b1fe523a92b: Status 404 returned error can't find the container with id 0ae3b2f51b5b1c82263a97d16108ebea631fe33ff5b550bc99349b1fe523a92b Jul 16 13:40:21.886304 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:21.886275 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:22.083114 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.083038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:22.083114 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.083099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:22.083331 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.083197 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:22.083331 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.083259 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:23.083241567 +0000 UTC m=+3.031358258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:22.083331 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.083198 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:22.083331 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.083299 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:23.083289391 +0000 UTC m=+3.031406078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:22.184066 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.184032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:22.184296 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.184235 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:22.184296 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.184256 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:22.184296 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.184268 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:22.184529 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.184321 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:23.184303842 +0000 UTC m=+3.132420532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:22.298317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.298279 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:22.417289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.417152 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 13:35:21 +0000 UTC" deadline="2028-01-04 23:51:16.425157591 +0000 UTC" Jul 16 13:40:22.417289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.417213 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12898h10m54.007947713s" Jul 16 13:40:22.470056 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.469467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:22.470056 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:22.469636 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:22.475055 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.474974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqmpw" event={"ID":"01a5411a-ae3d-42a2-98cd-926bd4155ac8","Type":"ContainerStarted","Data":"0ae3b2f51b5b1c82263a97d16108ebea631fe33ff5b550bc99349b1fe523a92b"} Jul 16 13:40:22.476430 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.476350 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" event={"ID":"cff69847-6d3c-4063-b247-487a7a21d795","Type":"ContainerStarted","Data":"96e7e1a14493dba679979de45cce678568d1efbff2026a72887c6a71317eb347"} Jul 16 13:40:22.478706 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.478598 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"3488ceeb156234f3324302b3af5ecdc8155f7e06665027dab09711232ab892d1"} Jul 16 13:40:22.480404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.480349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jn7sh" event={"ID":"2234a173-1018-4893-968c-037f58da8dea","Type":"ContainerStarted","Data":"788f3ca9921a299142a80bad43b60df2398041545048721525b6f4fd91b6984f"} Jul 16 13:40:22.482655 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.482609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9c256" event={"ID":"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3","Type":"ContainerStarted","Data":"9b540b3396b82f867bbfa9922ddf272fa25a27dbbd3a55807ca79e9f82b3107e"} Jul 16 13:40:22.485484 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.485406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" event={"ID":"291f71208e98001156098a5810a8cb54","Type":"ContainerStarted","Data":"96d4f438b13d33a1ca00912bee8dabbc0453813d1ad965ea38d379c5598bd9ed"} Jul 16 13:40:22.489189 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.489106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerStarted","Data":"173b1f559dc12c86430e42c9353a5cdff6ea524c6bd43a6106e30254e017873f"} Jul 16 13:40:22.491687 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.491622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lb6pg" event={"ID":"2ef62460-d86f-49bc-9de3-3023d384570f","Type":"ContainerStarted","Data":"c2d4289bb998939def4ecbb19f55b39d42264a80232789538ad2288ef718bf3b"} Jul 16 13:40:22.494904 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.494863 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" event={"ID":"8f7287ee-f367-4541-b6b7-08b20a94971d","Type":"ContainerStarted","Data":"196932f17c106dfc8a03c7bf7fbf5826536447f22a6a4b13b45c67e285409f05"} Jul 16 13:40:22.496674 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:22.496651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f6wm" event={"ID":"8489bd65-432f-4e2e-8e58-bd4e2463d7a3","Type":"ContainerStarted","Data":"1813cdb8ab3a319f01dfecfa1254f5350114821fc724da9dd1b62dc09ada2fab"} Jul 16 13:40:23.091540 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.091502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:23.091741 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.091567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:23.091741 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.091703 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:23.091851 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.091767 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:25.091747739 +0000 UTC m=+5.039864444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:23.092189 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.092170 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:23.092272 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.092223 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:25.092209657 +0000 UTC m=+5.040326347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:23.192842 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.192803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:23.193047 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.193004 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:23.193047 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.193023 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:23.193047 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.193036 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:23.193206 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.193093 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:25.193076254 +0000 UTC m=+5.141192944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:23.377892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.377591 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:23.417623 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.417581 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 13:35:21 +0000 UTC" deadline="2027-12-25 17:33:56.792463907 +0000 UTC" Jul 16 13:40:23.417623 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.417619 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12651h53m33.374849668s" Jul 16 13:40:23.468186 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.468148 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:23.468350 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:23.468219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:23.468422 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.468361 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:23.468481 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:23.468443 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:24.091460 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:24.091257 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 13:40:24.472909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:24.472346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:24.472909 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:24.472494 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:25.111448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:25.111547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.111680 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.111750 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:29.111728854 +0000 UTC m=+9.059845559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.112231 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:25.112318 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.112283 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:29.112267297 +0000 UTC m=+9.060383987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:25.212823 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:25.212727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:25.212999 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.212948 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:25.212999 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.212970 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:25.212999 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.212984 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:25.213175 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.213043 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:29.213024372 +0000 UTC m=+9.161141060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:25.468583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:25.468336 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:25.468583 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.468466 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:25.469042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:25.468862 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:25.469042 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:25.468996 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:26.468946 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:26.468911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:26.469418 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:26.469067 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:27.468925 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:27.468468 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:27.468925 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:27.468614 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:27.469467 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:27.468468 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:27.469467 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:27.469077 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:28.470312 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:28.469956 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:28.470735 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:28.470333 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:28.512004 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:28.511930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" event={"ID":"8976d328a4aa8b80317cf76fe353706f","Type":"ContainerStarted","Data":"9bb227ca9903ecf7474c9daaa131607c9ba6a8e417a9587147f3cf3377a98a82"} Jul 16 13:40:29.142949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:29.142257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:29.142949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:29.142330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:29.142949 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.142475 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:29.142949 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.142552 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:37.142533647 +0000 UTC m=+17.090650333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:29.142949 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.142955 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:29.143357 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.143010 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:37.14299525 +0000 UTC m=+17.091111947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:29.243309 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:29.243261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:29.243476 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.243460 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:29.243556 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.243481 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:29.243556 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.243494 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:29.243655 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.243561 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:37.243543195 +0000 UTC m=+17.191659906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:29.469099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:29.468558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:29.469099 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.468753 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:29.469304 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:29.469105 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:29.469304 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:29.469213 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:30.469473 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:30.468999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:30.469473 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:30.469115 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:31.468708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:31.468637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:31.468967 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:31.468761 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:31.468967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:31.468806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:31.468967 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:31.468954 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:32.468391 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:32.468349 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:32.468859 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:32.468487 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:33.468885 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:33.468837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:33.469304 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:33.468837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:33.469304 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:33.468972 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:33.469304 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:33.469071 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:34.468078 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:34.468041 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:34.468267 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:34.468155 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:35.468144 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:35.468104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:35.468551 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:35.468104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:35.468551 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:35.468267 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:35.468551 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:35.468320 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:36.468242 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:36.468211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:36.468709 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:36.468319 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:37.202466 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:37.202434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:37.202749 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:37.202493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:37.202749 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.202554 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:37.202749 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.202627 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:40:53.202607764 +0000 UTC m=+33.150724468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:37.202749 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.202649 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:37.202749 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.202720 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:53.202704128 +0000 UTC m=+33.150820815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:37.303024 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:37.302980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:37.303228 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.303143 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:37.303228 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.303169 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:37.303228 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.303182 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:37.303365 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.303249 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:40:53.303228064 +0000 UTC m=+33.251344754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:37.468264 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:37.468157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:37.468704 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.468293 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:37.468704 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:37.468177 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:37.468704 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:37.468497 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:38.468604 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:38.468567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:38.469134 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:38.468701 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:39.467814 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:39.467776 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:39.468018 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:39.467790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:39.468018 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:39.467924 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:39.468018 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:39.467987 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:40.469209 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:40.469177 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:40.469622 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:40.469262 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:41.468075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.467922 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:41.468075 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:41.468037 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:41.468075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.468059 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:41.468269 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:41.468149 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:41.537679 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.537650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqmpw" event={"ID":"01a5411a-ae3d-42a2-98cd-926bd4155ac8","Type":"ContainerStarted","Data":"7dd956dc6c34b3b5d159aeea5dbaba220b4a309f970027812dee706b6fda979f"} Jul 16 13:40:41.539415 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.539387 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" event={"ID":"cff69847-6d3c-4063-b247-487a7a21d795","Type":"ContainerStarted","Data":"f2da1c1dcd664525217c0991cfa5db32643ffb24d303e4b79b46aac7c452b159"} Jul 16 13:40:41.541705 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.541684 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:40:41.542187 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.542105 2563 generic.go:356] "Generic (PLEG): container finished" podID="9e789ac3-b4c4-42d6-85e4-c0dad9024b5e" containerID="0bf522b25720d367e0ac73ffd47c302a8c41ef49eb26e913181a2b554e4a1b29" exitCode=1 Jul 16 13:40:41.542187 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.542128 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerDied","Data":"0bf522b25720d367e0ac73ffd47c302a8c41ef49eb26e913181a2b554e4a1b29"} Jul 16 13:40:41.542333 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.542200 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"0dbe0950a8c10364528a5c149c4c1679b9c73c3a1a5bf28db4d7f6ca16eb6fa2"} Jul 16 13:40:41.543849 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.543814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9c256" event={"ID":"9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3","Type":"ContainerStarted","Data":"a877da76518636e66704a30b502721951d0782867f0e88f33c8dde33b2946eea"} Jul 16 13:40:41.545455 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.545435 2563 generic.go:356] "Generic (PLEG): container finished" podID="291f71208e98001156098a5810a8cb54" containerID="dda9a1c722fb96933b17e495044cb00bd74edb4701f895ab7d483fd4bdb64e48" exitCode=0 Jul 16 13:40:41.545626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.545601 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" event={"ID":"291f71208e98001156098a5810a8cb54","Type":"ContainerDied","Data":"dda9a1c722fb96933b17e495044cb00bd74edb4701f895ab7d483fd4bdb64e48"} Jul 16 13:40:41.547790 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.547627 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="1707a579c680a8ac163975d86d9c151b15659f7b1df89452252a6aa9a6c281f9" exitCode=0 Jul 16 13:40:41.547944 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.547703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"1707a579c680a8ac163975d86d9c151b15659f7b1df89452252a6aa9a6c281f9"} Jul 16 13:40:41.550379 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.550062 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lb6pg" event={"ID":"2ef62460-d86f-49bc-9de3-3023d384570f","Type":"ContainerStarted","Data":"36d2bf9c4105c0e0963bf0401e0d078a97e9ffe15db8d3d078aee9d49516fc2f"} Jul 16 13:40:41.551892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.551836 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nqmpw" podStartSLOduration=2.219069 podStartE2EDuration="21.551823524s" totalImagesPullingTime="19.332754524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.551635233 +0000 UTC m=+21.499751942" watchObservedRunningTime="2026-07-16 13:40:41.551823524 +0000 UTC m=+21.499940230" Jul 16 13:40:41.551997 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.551939 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" event={"ID":"8f7287ee-f367-4541-b6b7-08b20a94971d","Type":"ContainerStarted","Data":"3b22ad5b805a2277a034ae8087343767b645b3833739cb6961257dc4dfc46df8"} Jul 16 13:40:41.553645 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.553617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f6wm" event={"ID":"8489bd65-432f-4e2e-8e58-bd4e2463d7a3","Type":"ContainerStarted","Data":"fd6ddd580cfb244f1ee3259e6e00724060585ae1d94ad61ce898165c8a8d93bf"} Jul 16 13:40:41.555400 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.555380 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" event={"ID":"8976d328a4aa8b80317cf76fe353706f","Type":"ContainerStarted","Data":"2ac0eb393552f78ec33c6f0cb47b71397c218f8323737bf7bf69c6a4fe8ec7fe"} Jul 16 13:40:41.586512 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.586455 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9c256" podStartSLOduration=2.201493987 podStartE2EDuration="21.58643312s" totalImagesPullingTime="19.384939133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.585543665 +0000 UTC m=+21.533660373" watchObservedRunningTime="2026-07-16 13:40:41.58643312 +0000 UTC m=+21.534549828" Jul 16 13:40:41.624220 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.624172 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lb6pg" podStartSLOduration=2.495074728 podStartE2EDuration="21.62413869s" totalImagesPullingTime="19.129063962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.623828908 +0000 UTC m=+21.571945617" watchObservedRunningTime="2026-07-16 13:40:41.62413869 +0000 UTC m=+21.572255398" Jul 16 13:40:41.640816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.640711 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8f6wm" podStartSLOduration=3.606373795 podStartE2EDuration="21.640696815s" totalImagesPullingTime="18.03432302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.640235214 +0000 UTC m=+21.588351925" watchObservedRunningTime="2026-07-16 13:40:41.640696815 +0000 UTC m=+21.588813521" Jul 16 13:40:41.656432 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.656388 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-14.ec2.internal" podStartSLOduration=20.656371228 podStartE2EDuration="20.656371228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.656237277 +0000 UTC m=+21.604353986" watchObservedRunningTime="2026-07-16 13:40:41.656371228 +0000 UTC m=+21.604487935" Jul 16 13:40:41.678503 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:41.678463 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dvvfh" podStartSLOduration=2.540114375 podStartE2EDuration="21.678445632s" totalImagesPullingTime="19.138331257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:41.678408466 +0000 UTC m=+21.626525167" watchObservedRunningTime="2026-07-16 13:40:41.678445632 +0000 UTC m=+21.626562316" Jul 16 13:40:42.472571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.472484 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:42.472748 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:42.472618 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:42.562293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.562263 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:40:42.562751 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.562703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"0ba21876c512b88449b7bfdbc2c501b185157bf017c16039bb53d340a61a6926"} Jul 16 13:40:42.562818 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.562794 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"a4aa07406bcf51f97c056bb4161ee7e2a32728b06920f4283ef6bdb19988e016"} Jul 16 13:40:42.562959 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.562926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"7a24dc04b1882a6cc225ad07736148ef0cfa4874a5f1456740b8509736678634"} Jul 16 13:40:42.563012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.562981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"c64c474bc8031b90f50c7e4a0c239686c32dbff6c4fd7e891b933f06eced1b36"} Jul 16 13:40:42.565509 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.565480 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" event={"ID":"291f71208e98001156098a5810a8cb54","Type":"ContainerStarted","Data":"3de7ac165d322fc2b5f6e9d3a70516f73c0b32a7f4acda1f622abae60847e8e3"} Jul 16 13:40:42.580163 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:42.580117 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-14.ec2.internal" podStartSLOduration=21.580103755 podStartE2EDuration="21.580103755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:42.579795448 +0000 UTC m=+22.527912157" watchObservedRunningTime="2026-07-16 13:40:42.580103755 +0000 UTC m=+22.528220461" Jul 16 13:40:43.392307 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.392153 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 13:40:43.438386 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.438282 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T13:40:43.392170604Z","UUID":"0a889167-132e-4b48-b9da-f2cbfc2d9907","Handler":null,"Name":"","Endpoint":""} Jul 16 13:40:43.440123 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.440100 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 13:40:43.440272 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.440145 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 13:40:43.468177 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.468140 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:43.468364 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.468185 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:43.468364 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:43.468253 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:43.468481 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:43.468417 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:43.569913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.569814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" event={"ID":"cff69847-6d3c-4063-b247-487a7a21d795","Type":"ContainerStarted","Data":"58fe2f44604fe32e68c1ea4a438cedd3b8f7bdf00683ec7d7a5ff531396a7db3"} Jul 16 13:40:43.571517 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.571483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jn7sh" event={"ID":"2234a173-1018-4893-968c-037f58da8dea","Type":"ContainerStarted","Data":"996ab6ce8efda1ab18391e86388e8853adf04751036d709c735ba05425c5a2da"} Jul 16 13:40:43.586028 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:43.585977 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jn7sh" podStartSLOduration=4.223167614 podStartE2EDuration="23.585957656s" totalImagesPullingTime="19.362790042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:43.585662936 +0000 UTC m=+23.533779644" watchObservedRunningTime="2026-07-16 13:40:43.585957656 +0000 UTC m=+23.534074364" Jul 16 13:40:44.472293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:44.472227 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:44.472451 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:44.472339 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:44.811537 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:44.811457 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:44.814572 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:44.812988 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:45.468268 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.468087 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:45.468444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.468118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:45.468444 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:45.468347 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:45.468444 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:45.468417 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:45.578762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.578737 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:40:45.579139 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.579113 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"59b2a8bd1245763ec78d674fdce376a6286e1d51332d980ded7a1201bd265b94"} Jul 16 13:40:45.579349 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.579336 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:45.580391 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:45.580373 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8f6wm" Jul 16 13:40:46.468441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:46.468377 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:46.469056 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:46.468471 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:46.583369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:46.583334 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="87c89826e979ca010d5ac0ef17fca5aefd4a0ccf100ff5e84f8fb5ee03220cc0" exitCode=0 Jul 16 13:40:46.583544 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:46.583418 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"87c89826e979ca010d5ac0ef17fca5aefd4a0ccf100ff5e84f8fb5ee03220cc0"} Jul 16 13:40:46.585710 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:46.585689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" event={"ID":"cff69847-6d3c-4063-b247-487a7a21d795","Type":"ContainerStarted","Data":"2c25470ceccfef56859a94a479899fef42fd3848e28cb32aa42a6e045e049a27"} Jul 16 13:40:46.622601 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:46.622552 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j22rp" podStartSLOduration=3.165392051 podStartE2EDuration="26.622535999s" totalImagesPullingTime="23.457143948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:46.622445536 +0000 UTC m=+26.570562243" watchObservedRunningTime="2026-07-16 13:40:46.622535999 +0000 UTC m=+26.570652706" Jul 16 13:40:47.467821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:47.467794 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:47.468008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:47.467794 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:47.468008 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:47.467903 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:47.468008 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:47.467971 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:48.468610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.468427 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:48.469203 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:48.468704 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:48.592553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.592467 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="d33d00835a1af08f575d34bb4495e48a4f546df91b8fd8dee703408eb23c9a01" exitCode=0 Jul 16 13:40:48.592553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.592512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"d33d00835a1af08f575d34bb4495e48a4f546df91b8fd8dee703408eb23c9a01"} Jul 16 13:40:48.596207 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.596176 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:40:48.596564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.596531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"95ecde5e73332b0a1e793d56ff8287cc32009352dcd59b57e6df4183e157e1e1"} Jul 16 13:40:48.596837 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.596822 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:48.596909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.596890 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:48.597030 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.597013 2563 scope.go:117] "RemoveContainer" containerID="0bf522b25720d367e0ac73ffd47c302a8c41ef49eb26e913181a2b554e4a1b29" Jul 16 13:40:48.613902 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:48.613860 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:49.468477 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.468428 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:49.468477 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.468458 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:49.468676 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:49.468584 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:49.468943 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:49.468691 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:49.608399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.608374 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:40:49.608780 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.608757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" event={"ID":"9e789ac3-b4c4-42d6-85e4-c0dad9024b5e","Type":"ContainerStarted","Data":"51116fcd28e330f1cd5fed1c1e0b1162d6d730103464bce13d3300e210c6ae60"} Jul 16 13:40:49.609190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.609140 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:49.629247 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.629221 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:40:49.639908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.639844 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" podStartSLOduration=10.573620122 podStartE2EDuration="29.639828946s" totalImagesPullingTime="19.066208824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:49.638592822 +0000 UTC m=+29.586709554" watchObservedRunningTime="2026-07-16 13:40:49.639828946 +0000 UTC m=+29.587945652" Jul 16 13:40:49.758277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.758197 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8t8z"] Jul 16 13:40:49.758422 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.758304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:49.758422 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:49.758392 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:49.761380 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.761357 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9glr"] Jul 16 13:40:49.761533 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.761465 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:49.761617 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:49.761570 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:49.771762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.771734 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kf4n6"] Jul 16 13:40:49.771946 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:49.771837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:49.772015 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:49.771955 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:50.613709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:50.613463 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="d321f35f01f6f78ed2ee8c52fa4f8ec1b50dd8c2f62c2ef339dcbc9a6c4aad61" exitCode=0 Jul 16 13:40:50.613709 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:50.613551 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"d321f35f01f6f78ed2ee8c52fa4f8ec1b50dd8c2f62c2ef339dcbc9a6c4aad61"} Jul 16 13:40:51.468271 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:51.468240 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:51.468441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:51.468346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:51.468441 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:51.468360 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:51.468441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:51.468379 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:51.468585 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:51.468464 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:51.468585 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:51.468554 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:53.219223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.219186 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:53.219926 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.219243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:53.219926 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.219364 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:53.219926 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.219444 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret podName:9d7cc4eb-0314-42c9-ae8d-39152296fbda nodeName:}" failed. No retries permitted until 2026-07-16 13:41:25.219425572 +0000 UTC m=+65.167542274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret") pod "global-pull-secret-syncer-n9glr" (UID: "9d7cc4eb-0314-42c9-ae8d-39152296fbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 13:40:53.219926 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.219365 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:53.219926 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.219509 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:25.2194971 +0000 UTC m=+65.167613784 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 13:40:53.320501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.320454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:53.320718 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.320624 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 13:40:53.320718 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.320644 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 13:40:53.320718 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.320657 2563 projected.go:196] Error preparing data for projected volume kube-api-access-b225t for pod openshift-network-diagnostics/network-check-target-g8t8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:53.320896 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.320739 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t podName:443ebb9f-5c96-4468-b2b1-e94275ae577e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:25.320719534 +0000 UTC m=+65.268836225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b225t" (UniqueName: "kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t") pod "network-check-target-g8t8z" (UID: "443ebb9f-5c96-4468-b2b1-e94275ae577e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 13:40:53.468203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.468165 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:53.468203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.468207 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:53.468439 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.468216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:53.468439 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.468300 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8t8z" podUID="443ebb9f-5c96-4468-b2b1-e94275ae577e" Jul 16 13:40:53.468439 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.468387 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kf4n6" podUID="8c18e82a-f845-4475-bc78-f29acf434c83" Jul 16 13:40:53.468590 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:53.468478 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9glr" podUID="9d7cc4eb-0314-42c9-ae8d-39152296fbda" Jul 16 13:40:53.854810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.854781 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-14.ec2.internal" event="NodeReady" Jul 16 13:40:53.855131 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.854956 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 13:40:53.890653 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.890618 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:40:53.894479 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.894451 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:53.897019 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.896996 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 13:40:53.898079 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.898059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xfh2z\"" Jul 16 13:40:53.898196 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.898059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 13:40:53.898444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.898383 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 13:40:53.903869 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.903845 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 13:40:53.910074 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.910050 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:40:53.914016 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.913963 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-252xq"] Jul 16 13:40:53.916976 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.916956 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rrd8j"] Jul 16 13:40:53.917108 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.917090 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:53.919527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.919503 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 13:40:53.919656 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.919570 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 13:40:53.919656 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.919509 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 13:40:53.919810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.919657 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6q5v4\"" Jul 16 13:40:53.919977 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.919961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:53.922904 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.922865 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 13:40:53.923021 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.922866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfg5g\"" Jul 16 13:40:53.923084 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.923049 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 13:40:53.928279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.928259 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-252xq"] Jul 16 13:40:53.932092 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:53.931780 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrd8j"] Jul 16 13:40:54.026849 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.026770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.026849 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.026807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbblf\" (UniqueName: \"kubernetes.io/projected/48a1bdd5-e23f-4f26-947f-61e7965cad0c-kube-api-access-tbblf\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.026849 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.026833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3488b5d2-1e52-4605-91b3-7a0248cffd18-config-volume\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.027106 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.026920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027106 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.026971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48a1bdd5-e23f-4f26-947f-61e7965cad0c-cert\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.027106 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027042 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027106 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlchf\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.027261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blghs\" (UniqueName: \"kubernetes.io/projected/3488b5d2-1e52-4605-91b3-7a0248cffd18-kube-api-access-blghs\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.027416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027314 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.027416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.027339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488b5d2-1e52-4605-91b3-7a0248cffd18-tmp-dir\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.052694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.052657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:40:54.127708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.127666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48a1bdd5-e23f-4f26-947f-61e7965cad0c-cert\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.127927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.127747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.127927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.127779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.127927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.127822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.127944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlchf\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blghs\" (UniqueName: \"kubernetes.io/projected/3488b5d2-1e52-4605-91b3-7a0248cffd18-kube-api-access-blghs\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128109 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488b5d2-1e52-4605-91b3-7a0248cffd18-tmp-dir\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.128130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128150 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbblf\" (UniqueName: \"kubernetes.io/projected/48a1bdd5-e23f-4f26-947f-61e7965cad0c-kube-api-access-tbblf\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3488b5d2-1e52-4605-91b3-7a0248cffd18-config-volume\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128201 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.128295 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:40:54.128459 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.128411 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:54.628388023 +0000 UTC m=+34.576504714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:40:54.128770 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3488b5d2-1e52-4605-91b3-7a0248cffd18-tmp-dir\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.128770 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.128706 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:40:54.128770 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.128730 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:40:54.128933 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.128810 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:54.62878563 +0000 UTC m=+34.576902341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:40:54.128933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.129048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.128973 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3488b5d2-1e52-4605-91b3-7a0248cffd18-config-volume\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.129257 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.129235 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.132741 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.132720 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.132896 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.132739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.132896 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.132771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48a1bdd5-e23f-4f26-947f-61e7965cad0c-cert\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.136550 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.136524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:40:54.137946 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.137921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbblf\" (UniqueName: \"kubernetes.io/projected/48a1bdd5-e23f-4f26-947f-61e7965cad0c-kube-api-access-tbblf\") pod \"ingress-canary-252xq\" (UID: \"48a1bdd5-e23f-4f26-947f-61e7965cad0c\") " pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.138121 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.138101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.141076 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.138333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlchf\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.141076 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.138437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blghs\" (UniqueName: \"kubernetes.io/projected/3488b5d2-1e52-4605-91b3-7a0248cffd18-kube-api-access-blghs\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.229649 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.229609 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-252xq" Jul 16 13:40:54.631520 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.631476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:54.631721 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.631537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:54.631721 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.631650 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:40:54.631721 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.631671 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:40:54.631920 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.631647 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:40:54.631920 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.631749 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:55.631727452 +0000 UTC m=+35.579844187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:40:54.631920 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:54.631793 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:55.631780195 +0000 UTC m=+35.579896895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:40:54.792156 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:54.792000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-252xq"] Jul 16 13:40:54.795242 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:40:54.795215 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a1bdd5_e23f_4f26_947f_61e7965cad0c.slice/crio-40e88003ee5d3d2dc684bfe029af7459c44d0a655b98bfb67be8c1040f60155d WatchSource:0}: Error finding container 40e88003ee5d3d2dc684bfe029af7459c44d0a655b98bfb67be8c1040f60155d: Status 404 returned error can't find the container with id 40e88003ee5d3d2dc684bfe029af7459c44d0a655b98bfb67be8c1040f60155d Jul 16 13:40:55.468616 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.468528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:40:55.469204 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.468528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:40:55.469204 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.468528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:40:55.471360 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.471337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 13:40:55.471494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.471403 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 13:40:55.472150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.472123 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6cxrw\"" Jul 16 13:40:55.472150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.472143 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 13:40:55.472314 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.472154 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 13:40:55.472314 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.472160 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6lhsq\"" Jul 16 13:40:55.628935 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.628899 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="18f07a956d2deb3cae398e4692960daffda248d37c4f742f880f838e813fc71d" exitCode=0 Jul 16 13:40:55.629120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.628974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"18f07a956d2deb3cae398e4692960daffda248d37c4f742f880f838e813fc71d"} Jul 16 13:40:55.630332 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.630304 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-252xq" event={"ID":"48a1bdd5-e23f-4f26-947f-61e7965cad0c","Type":"ContainerStarted","Data":"40e88003ee5d3d2dc684bfe029af7459c44d0a655b98bfb67be8c1040f60155d"} Jul 16 13:40:55.639240 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.639217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:55.639327 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:55.639260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:55.639396 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:55.639370 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:40:55.639448 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:55.639428 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:57.63940691 +0000 UTC m=+37.587523607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:40:55.639505 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:55.639461 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:40:55.639505 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:55.639481 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:40:55.639595 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:55.639532 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:40:57.639514148 +0000 UTC m=+37.587630853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:40:56.636573 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:56.636532 2563 generic.go:356] "Generic (PLEG): container finished" podID="70c04af3-de33-48d5-9959-2b6c5fd45515" containerID="b0459ade0997fcee6939aac75686b5c3083494a4582945b3f297279cb2be6de4" exitCode=0 Jul 16 13:40:56.637049 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:56.636605 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerDied","Data":"b0459ade0997fcee6939aac75686b5c3083494a4582945b3f297279cb2be6de4"} Jul 16 13:40:57.640938 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.640618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-252xq" event={"ID":"48a1bdd5-e23f-4f26-947f-61e7965cad0c","Type":"ContainerStarted","Data":"1d8fed23f8324f3b66dbe1aa230e695762b06b1bc8c7e4e37d08f0715afb1594"} Jul 16 13:40:57.644130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.644101 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt66x" event={"ID":"70c04af3-de33-48d5-9959-2b6c5fd45515","Type":"ContainerStarted","Data":"f54b36a0b0b6e769d2fe5dd216abf25042156c1c02bc50bae6644c8313996704"} Jul 16 13:40:57.655678 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.655652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:40:57.655817 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.655685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:40:57.655817 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:57.655793 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:40:57.655939 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:57.655848 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:01.655835977 +0000 UTC m=+41.603952667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:40:57.655939 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:57.655797 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:40:57.655939 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:57.655897 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:40:57.655939 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:40:57.655940 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:01.655928927 +0000 UTC m=+41.604045611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:40:57.656905 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.656814 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-252xq" podStartSLOduration=2.016780758 podStartE2EDuration="4.656797582s" totalImagesPullingTime="2.640016824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:57.656459454 +0000 UTC m=+37.604576164" watchObservedRunningTime="2026-07-16 13:40:57.656797582 +0000 UTC m=+37.604914291" Jul 16 13:40:57.677619 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.677566 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kt66x" podStartSLOduration=8.1337052 podStartE2EDuration="37.677544115s" totalImagesPullingTime="29.543838915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:40:57.675943022 +0000 UTC m=+37.624059751" watchObservedRunningTime="2026-07-16 13:40:57.677544115 +0000 UTC m=+37.625660803" Jul 16 13:40:57.900110 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:57.900005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:40:59.509405 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:59.509377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:40:59.890126 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:59.890077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:40:59.992775 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:40:59.992739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:00.165934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:00.165800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:00.255764 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:00.255718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:00.356198 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:00.356166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:00.457303 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:00.457214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:01.193113 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:01.193077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:01.545171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:01.545142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:01.644067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:01.644010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:01.690495 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:01.690465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:01.690668 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:01.690500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:01.690668 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:01.690626 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:41:01.690738 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:01.690687 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:09.690671309 +0000 UTC m=+49.638787997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:41:01.690738 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:01.690634 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:41:01.690738 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:01.690729 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:41:01.690849 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:01.690774 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:09.690761773 +0000 UTC m=+49.638878458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:41:03.975554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:03.975496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:04.847730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:04.847699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:09.412637 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:09.412597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:09.476098 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:09.476062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:09.748809 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:09.748712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:09.749042 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:09.748868 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:41:09.749042 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:09.748911 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:41:09.749042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:09.748960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:09.749042 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:09.748978 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:25.748956545 +0000 UTC m=+65.697073243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:41:09.749042 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:09.749041 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:41:09.749227 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:09.749075 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:25.749067845 +0000 UTC m=+65.697184533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:41:10.473440 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:10.473404 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:11.926690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:11.926655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:12.028020 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:12.027987 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:14.927693 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:14.927659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:14.982298 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:14.982268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:16.124805 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:16.124766 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:16.327526 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:16.327497 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:16.392315 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:16.392227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:18.524065 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:18.524023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:18.580718 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:18.580691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:21.626410 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:21.626382 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltrtt" Jul 16 13:41:21.788114 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:21.788086 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:21.848210 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:21.848180 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:25.261333 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.261297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:41:25.261766 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.261491 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:41:25.263628 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.263608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 13:41:25.263740 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.263610 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 13:41:25.272336 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.272312 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 13:41:25.272434 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.272381 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs podName:8c18e82a-f845-4475-bc78-f29acf434c83 nodeName:}" failed. No retries permitted until 2026-07-16 13:42:29.272363357 +0000 UTC m=+129.220480050 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs") pod "network-metrics-daemon-kf4n6" (UID: "8c18e82a-f845-4475-bc78-f29acf434c83") : secret "metrics-daemon-secret" not found Jul 16 13:41:25.276198 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.276174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7cc4eb-0314-42c9-ae8d-39152296fbda-original-pull-secret\") pod \"global-pull-secret-syncer-n9glr\" (UID: \"9d7cc4eb-0314-42c9-ae8d-39152296fbda\") " pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:41:25.362854 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.362822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:41:25.365296 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.365265 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 13:41:25.376008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.375986 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 13:41:25.387228 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.387204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b225t\" (UniqueName: \"kubernetes.io/projected/443ebb9f-5c96-4468-b2b1-e94275ae577e-kube-api-access-b225t\") pod \"network-check-target-g8t8z\" (UID: \"443ebb9f-5c96-4468-b2b1-e94275ae577e\") " pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:41:25.488204 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.488168 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9glr" Jul 16 13:41:25.497706 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.497233 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6lhsq\"" Jul 16 13:41:25.504762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.504744 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:41:25.645948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.645909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9glr"] Jul 16 13:41:25.649464 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:25.649437 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7cc4eb_0314_42c9_ae8d_39152296fbda.slice/crio-a0192ca6b626ff289c78403a05347c80bcfc0bbbbeecce4abe078133d072c9ae WatchSource:0}: Error finding container a0192ca6b626ff289c78403a05347c80bcfc0bbbbeecce4abe078133d072c9ae: Status 404 returned error can't find the container with id a0192ca6b626ff289c78403a05347c80bcfc0bbbbeecce4abe078133d072c9ae Jul 16 13:41:25.662552 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.662531 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8t8z"] Jul 16 13:41:25.665654 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:25.665632 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443ebb9f_5c96_4468_b2b1_e94275ae577e.slice/crio-6f1ecb530d52769f189b2b6b7b08d697520167df541f5ffd5e1a267ab704eb43 WatchSource:0}: Error finding container 6f1ecb530d52769f189b2b6b7b08d697520167df541f5ffd5e1a267ab704eb43: Status 404 returned error can't find the container with id 6f1ecb530d52769f189b2b6b7b08d697520167df541f5ffd5e1a267ab704eb43 Jul 16 13:41:25.711412 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.711378 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8t8z" event={"ID":"443ebb9f-5c96-4468-b2b1-e94275ae577e","Type":"ContainerStarted","Data":"6f1ecb530d52769f189b2b6b7b08d697520167df541f5ffd5e1a267ab704eb43"} Jul 16 13:41:25.712322 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.712297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9glr" event={"ID":"9d7cc4eb-0314-42c9-ae8d-39152296fbda","Type":"ContainerStarted","Data":"a0192ca6b626ff289c78403a05347c80bcfc0bbbbeecce4abe078133d072c9ae"} Jul 16 13:41:25.769062 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.768988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:25.769062 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:25.769029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:25.769198 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.769170 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 13:41:25.769231 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.769194 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 13:41:25.769231 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.769210 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cc4c6c8f-t5l5z: secret "image-registry-tls" not found Jul 16 13:41:25.769231 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.769230 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls podName:3488b5d2-1e52-4605-91b3-7a0248cffd18 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:57.769215168 +0000 UTC m=+97.717331854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls") pod "dns-default-rrd8j" (UID: "3488b5d2-1e52-4605-91b3-7a0248cffd18") : secret "dns-default-metrics-tls" not found Jul 16 13:41:25.769328 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:25.769261 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls podName:998e8a1b-eb66-46e9-a3f6-bf996d21c932 nodeName:}" failed. No retries permitted until 2026-07-16 13:41:57.769249645 +0000 UTC m=+97.717366331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls") pod "image-registry-75cc4c6c8f-t5l5z" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932") : secret "image-registry-tls" not found Jul 16 13:41:31.729996 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:31.729953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9glr" event={"ID":"9d7cc4eb-0314-42c9-ae8d-39152296fbda","Type":"ContainerStarted","Data":"89f1c0873b296f042c255dd6b2990ec65ba23b05e99b6eec7a5cc53353696fb5"} Jul 16 13:41:31.731394 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:31.731373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8t8z" event={"ID":"443ebb9f-5c96-4468-b2b1-e94275ae577e","Type":"ContainerStarted","Data":"0b1b9da91ba369eac54df299526dbf639050c62ff1ab1b427de194d1c81d6ffd"} Jul 16 13:41:31.731505 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:31.731494 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:41:31.747717 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:31.747666 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n9glr" podStartSLOduration=66.426960146 podStartE2EDuration="1m11.747651402s" totalImagesPullingTime="5.320691256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:41:31.746590495 +0000 UTC m=+71.694707214" watchObservedRunningTime="2026-07-16 13:41:31.747651402 +0000 UTC m=+71.695768108" Jul 16 13:41:31.770260 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:31.770210 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g8t8z" podStartSLOduration=66.466199603 podStartE2EDuration="1m11.770194925s" totalImagesPullingTime="5.303995322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:41:31.769341175 +0000 UTC m=+71.717457882" watchObservedRunningTime="2026-07-16 13:41:31.770194925 +0000 UTC m=+71.718311610" Jul 16 13:41:32.152948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.152914 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5845f855c8-qr8xb"] Jul 16 13:41:32.157583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.157562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.161710 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.161687 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 13:41:32.161833 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.161689 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 13:41:32.161833 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.161805 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 13:41:32.162069 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.162055 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 13:41:32.162639 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.162612 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6khmq\"" Jul 16 13:41:32.162639 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.162617 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 13:41:32.162785 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.162665 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 13:41:32.186887 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.186843 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5845f855c8-qr8xb"] Jul 16 13:41:32.222094 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.222064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-stats-auth\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.222280 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.222104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.222280 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.222126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2kl\" (UniqueName: \"kubernetes.io/projected/a3383b24-2dfc-45d3-9480-855b6e22b92e-kube-api-access-5f2kl\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.222280 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.222244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.222422 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.222305 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-default-certificate\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.322861 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.322826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-default-certificate\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.323041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.322887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-stats-auth\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.323041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.322917 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.323041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.322933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2kl\" (UniqueName: \"kubernetes.io/projected/a3383b24-2dfc-45d3-9480-855b6e22b92e-kube-api-access-5f2kl\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.323041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.322969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.323208 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.323051 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 13:41:32.323208 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.323099 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:32.823081298 +0000 UTC m=+72.771198000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : configmap references non-existent config key: service-ca.crt Jul 16 13:41:32.323208 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.323127 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:32.823108515 +0000 UTC m=+72.771225216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : secret "router-metrics-certs-default" not found Jul 16 13:41:32.325277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.325246 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-stats-auth\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.325387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.325310 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-default-certificate\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.331952 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.331933 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2kl\" (UniqueName: \"kubernetes.io/projected/a3383b24-2dfc-45d3-9480-855b6e22b92e-kube-api-access-5f2kl\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.407706 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.407616 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:32.487544 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.487503 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:32.826320 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.826287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.826710 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:32.826343 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:32.826710 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.826456 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 13:41:32.826710 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.826524 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:33.826501541 +0000 UTC m=+73.774618241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : configmap references non-existent config key: service-ca.crt Jul 16 13:41:32.826710 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:32.826555 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:33.826543162 +0000 UTC m=+73.774659846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : secret "router-metrics-certs-default" not found Jul 16 13:41:33.834501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:33.834463 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:33.834913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:33.834523 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:33.834913 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:33.834617 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 13:41:33.834913 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:33.834654 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:35.834641719 +0000 UTC m=+75.782758404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : configmap references non-existent config key: service-ca.crt Jul 16 13:41:33.834913 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:33.834677 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:35.834663942 +0000 UTC m=+75.782780628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : secret "router-metrics-certs-default" not found Jul 16 13:41:34.560425 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:34.560378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:34.617265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:34.617228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:35.849465 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:35.849424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:35.849849 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:35.849490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:35.849849 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:35.849620 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:39.849597752 +0000 UTC m=+79.797714440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : configmap references non-existent config key: service-ca.crt Jul 16 13:41:35.849849 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:35.849628 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 13:41:35.849849 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:35.849684 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:39.849670817 +0000 UTC m=+79.797787502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : secret "router-metrics-certs-default" not found Jul 16 13:41:36.608384 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:36.608348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:36.668644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:36.668607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.156199 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.156164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.249528 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.249490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.346499 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.346470 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gghzs"] Jul 16 13:41:37.350349 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.350333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.352763 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.352741 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 13:41:37.353308 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.353290 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 13:41:37.353472 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.353456 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 13:41:37.353730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.353712 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zzlst\"" Jul 16 13:41:37.354072 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.354057 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 13:41:37.361622 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.361601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gghzs"] Jul 16 13:41:37.408984 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.408891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.461988 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.461956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74c1a604-b97f-4033-919d-6af5f3c9993c-data-volume\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.462151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.462030 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.462151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.462088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74c1a604-b97f-4033-919d-6af5f3c9993c-crio-socket\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.462151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.462104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.462151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.462119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftq4t\" (UniqueName: \"kubernetes.io/projected/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-api-access-ftq4t\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.482396 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.482368 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.562609 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74c1a604-b97f-4033-919d-6af5f3c9993c-crio-socket\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562609 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftq4t\" (UniqueName: \"kubernetes.io/projected/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-api-access-ftq4t\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74c1a604-b97f-4033-919d-6af5f3c9993c-data-volume\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562690 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74c1a604-b97f-4033-919d-6af5f3c9993c-crio-socket\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:37.562716 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.562772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.562812 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:37.562784 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:38.062764177 +0000 UTC m=+78.010880864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : configmap "kube-rbac-proxy" not found Jul 16 13:41:37.563067 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:37.562841 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 13:41:37.563067 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:37.562913 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:38.062897272 +0000 UTC m=+78.011013957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : secret "insights-runtime-extractor-tls" not found Jul 16 13:41:37.563067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.563043 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74c1a604-b97f-4033-919d-6af5f3c9993c-data-volume\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.574813 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.574792 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftq4t\" (UniqueName: \"kubernetes.io/projected/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-api-access-ftq4t\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:37.683361 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.683278 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.784418 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.784384 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.870561 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.870524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:37.934466 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:37.934378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:38.066940 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.066901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:38.067133 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.066999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:38.067133 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:38.067036 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 13:41:38.067133 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:38.067094 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 13:41:38.067133 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:38.067115 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:39.067094076 +0000 UTC m=+79.015210761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : configmap "kube-rbac-proxy" not found Jul 16 13:41:38.067361 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:38.067143 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:39.067131329 +0000 UTC m=+79.015248015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : secret "insights-runtime-extractor-tls" not found Jul 16 13:41:38.148442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.148407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:38.155395 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.155366 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt"] Jul 16 13:41:38.159054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.159035 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" Jul 16 13:41:38.161248 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.161233 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2vfk5\"" Jul 16 13:41:38.161248 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.161235 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 13:41:38.162039 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.162023 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 13:41:38.169121 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.169100 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt"] Jul 16 13:41:38.230354 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.230268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:38.267910 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.267857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slpg\" (UniqueName: \"kubernetes.io/projected/d790fecd-6593-4af5-8bb5-f5d17280ea1b-kube-api-access-5slpg\") pod \"migrator-7f59b54f6c-7klwt\" (UID: \"d790fecd-6593-4af5-8bb5-f5d17280ea1b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" Jul 16 13:41:38.369182 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.369121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slpg\" (UniqueName: \"kubernetes.io/projected/d790fecd-6593-4af5-8bb5-f5d17280ea1b-kube-api-access-5slpg\") pod \"migrator-7f59b54f6c-7klwt\" (UID: \"d790fecd-6593-4af5-8bb5-f5d17280ea1b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" Jul 16 13:41:38.377120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.377091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slpg\" (UniqueName: \"kubernetes.io/projected/d790fecd-6593-4af5-8bb5-f5d17280ea1b-kube-api-access-5slpg\") pod \"migrator-7f59b54f6c-7klwt\" (UID: \"d790fecd-6593-4af5-8bb5-f5d17280ea1b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" Jul 16 13:41:38.467785 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.467746 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" Jul 16 13:41:38.590511 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.590480 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt"] Jul 16 13:41:38.593722 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:38.593694 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd790fecd_6593_4af5_8bb5_f5d17280ea1b.slice/crio-42c7487ff9072a85a04a7a0dd1309d52c41fb5edfa516480fe93bedef8a06ccf WatchSource:0}: Error finding container 42c7487ff9072a85a04a7a0dd1309d52c41fb5edfa516480fe93bedef8a06ccf: Status 404 returned error can't find the container with id 42c7487ff9072a85a04a7a0dd1309d52c41fb5edfa516480fe93bedef8a06ccf Jul 16 13:41:38.638541 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.638501 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:38.697618 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.697585 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:38.748829 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:38.748793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" event={"ID":"d790fecd-6593-4af5-8bb5-f5d17280ea1b","Type":"ContainerStarted","Data":"42c7487ff9072a85a04a7a0dd1309d52c41fb5edfa516480fe93bedef8a06ccf"} Jul 16 13:41:39.030514 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.030484 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:39.075243 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.075201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:39.075412 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.075325 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 13:41:39.075412 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.075405 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:41.075383118 +0000 UTC m=+81.023499808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : configmap "kube-rbac-proxy" not found Jul 16 13:41:39.075517 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.075466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:39.075622 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.075604 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 13:41:39.075678 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.075672 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:41.075655085 +0000 UTC m=+81.023771774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : secret "insights-runtime-extractor-tls" not found Jul 16 13:41:39.131534 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.131492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:39.542624 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.542585 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lb6pg_2ef62460-d86f-49bc-9de3-3023d384570f/dns-node-resolver/0.log" Jul 16 13:41:39.726915 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.726884 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:39.754838 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.754803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" event={"ID":"d790fecd-6593-4af5-8bb5-f5d17280ea1b","Type":"ContainerStarted","Data":"9c8f1307e49fe12d844e9a08cecbe6b7d13404243f6dff1887dbe8515ce1d28a"} Jul 16 13:41:39.755011 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.754898 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" event={"ID":"d790fecd-6593-4af5-8bb5-f5d17280ea1b","Type":"ContainerStarted","Data":"dfa7165089f70dfd4de141313127987bb6263ea50714b5cf9d40a407cf134a3f"} Jul 16 13:41:39.770312 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.770261 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7klwt" podStartSLOduration=0.748995116 podStartE2EDuration="1.770246264s" totalImagesPullingTime="1.021251148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:41:39.769849241 +0000 UTC m=+79.717965982" watchObservedRunningTime="2026-07-16 13:41:39.770246264 +0000 UTC m=+79.718362971" Jul 16 13:41:39.791944 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.791912 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:39.883324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.883239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:39.883324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.883298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:39.883611 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.883374 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 13:41:39.883611 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.883434 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:47.883419015 +0000 UTC m=+87.831535703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : secret "router-metrics-certs-default" not found Jul 16 13:41:39.883611 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:39.883449 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle podName:a3383b24-2dfc-45d3-9480-855b6e22b92e nodeName:}" failed. No retries permitted until 2026-07-16 13:41:47.883442712 +0000 UTC m=+87.831559396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle") pod "router-default-5845f855c8-qr8xb" (UID: "a3383b24-2dfc-45d3-9480-855b6e22b92e") : configmap references non-existent config key: service-ca.crt Jul 16 13:41:39.893276 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:39.893235 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:40.542634 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:40.542601 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqmpw_01a5411a-ae3d-42a2-98cd-926bd4155ac8/node-ca/0.log" Jul 16 13:41:40.644804 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:40.644740 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:40.746491 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:40.746459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:40.824899 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:40.824794 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:40.942659 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:40.942629 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-252xq_48a1bdd5-e23f-4f26-947f-61e7965cad0c/serve-healthcheck-canary/0.log" Jul 16 13:41:41.094073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.093969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:41.094073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.094032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:41.094297 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:41.094102 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 13:41:41.094297 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:41.094141 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 13:41:41.094297 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:41.094154 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:45.094140413 +0000 UTC m=+85.042257098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : configmap "kube-rbac-proxy" not found Jul 16 13:41:41.094297 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:41.094217 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:45.094199605 +0000 UTC m=+85.042316290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : secret "insights-runtime-extractor-tls" not found Jul 16 13:41:41.245824 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.245784 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:41.482643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.482550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:41.561428 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.561393 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:41.707137 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.707083 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:41.806649 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.806609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:41.952141 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:41.952104 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:42.527602 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:42.527573 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:43.226356 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:43.226319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:43.326267 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:43.326223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:43.534911 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:43.534857 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:43.738082 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:43.738043 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:43.932468 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:43.932371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:44.549810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:44.549772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:44.932029 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:44.931942 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:45.020610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.020571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:45.128517 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.128481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:45.128661 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.128532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:45.128661 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:45.128631 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 13:41:45.128661 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:45.128635 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 13:41:45.128782 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:45.128686 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:53.12867103 +0000 UTC m=+93.076787715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : configmap "kube-rbac-proxy" not found Jul 16 13:41:45.128782 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:41:45.128698 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls podName:74c1a604-b97f-4033-919d-6af5f3c9993c nodeName:}" failed. No retries permitted until 2026-07-16 13:41:53.128692026 +0000 UTC m=+93.076808711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gghzs" (UID: "74c1a604-b97f-4033-919d-6af5f3c9993c") : secret "insights-runtime-extractor-tls" not found Jul 16 13:41:45.338416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.338383 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:45.542170 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.542133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:45.737655 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.737569 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:45.937852 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:45.937814 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:46.931127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:46.931093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:47.203202 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.203087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:47.341753 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.341720 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:47.540155 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.540128 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:47.727486 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.727452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:47.952414 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.952315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:47.952414 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.952380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:47.952984 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.952966 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3383b24-2dfc-45d3-9480-855b6e22b92e-service-ca-bundle\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:47.954643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:47.954622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3383b24-2dfc-45d3-9480-855b6e22b92e-metrics-certs\") pod \"router-default-5845f855c8-qr8xb\" (UID: \"a3383b24-2dfc-45d3-9480-855b6e22b92e\") " pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:48.066144 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.066083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:48.127740 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.127704 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:48.195629 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.195598 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5845f855c8-qr8xb"] Jul 16 13:41:48.198155 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:48.198127 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3383b24_2dfc_45d3_9480_855b6e22b92e.slice/crio-64d624517a7dbcbe6cfe50728f615c414852661fea18b4d7e497ca04133b3ccc WatchSource:0}: Error finding container 64d624517a7dbcbe6cfe50728f615c414852661fea18b4d7e497ca04133b3ccc: Status 404 returned error can't find the container with id 64d624517a7dbcbe6cfe50728f615c414852661fea18b4d7e497ca04133b3ccc Jul 16 13:41:48.728107 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.728074 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:48.786268 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.786235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5845f855c8-qr8xb" event={"ID":"a3383b24-2dfc-45d3-9480-855b6e22b92e","Type":"ContainerStarted","Data":"2157768fa5bc9c3f8186cfc014deeaf18e5439e2066d4b57b8580efd53b5a0d3"} Jul 16 13:41:48.786397 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.786285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5845f855c8-qr8xb" event={"ID":"a3383b24-2dfc-45d3-9480-855b6e22b92e","Type":"ContainerStarted","Data":"64d624517a7dbcbe6cfe50728f615c414852661fea18b4d7e497ca04133b3ccc"} Jul 16 13:41:48.806013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:48.805961 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5845f855c8-qr8xb" podStartSLOduration=16.805944547 podStartE2EDuration="16.805944547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:41:48.804439309 +0000 UTC m=+88.752556013" watchObservedRunningTime="2026-07-16 13:41:48.805944547 +0000 UTC m=+88.754061253" Jul 16 13:41:49.066460 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.066427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:49.072758 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.072729 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:49.126194 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.126158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:49.531175 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.531140 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:49.745262 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.745227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:49.789514 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.789443 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:49.790660 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:49.790640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5845f855c8-qr8xb" Jul 16 13:41:50.146225 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:50.146146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:50.293037 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:50.293002 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:50.730527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:50.730486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:51.127046 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:51.127011 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:51.329093 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:51.329054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:51.655864 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:51.655826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:51.754227 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:51.754193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:51.930929 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:51.930822 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:53.193788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.193747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:53.194193 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.193800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:53.194281 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.194263 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74c1a604-b97f-4033-919d-6af5f3c9993c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:53.196094 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.196071 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74c1a604-b97f-4033-919d-6af5f3c9993c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gghzs\" (UID: \"74c1a604-b97f-4033-919d-6af5f3c9993c\") " pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:53.259463 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.259414 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gghzs" Jul 16 13:41:53.380389 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.380354 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gghzs"] Jul 16 13:41:53.383683 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:53.383656 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c1a604_b97f_4033_919d_6af5f3c9993c.slice/crio-66af5c0c10b418849a946dca62366aa6dc04f0596cd2c403a6c190757ed20ad3 WatchSource:0}: Error finding container 66af5c0c10b418849a946dca62366aa6dc04f0596cd2c403a6c190757ed20ad3: Status 404 returned error can't find the container with id 66af5c0c10b418849a946dca62366aa6dc04f0596cd2c403a6c190757ed20ad3 Jul 16 13:41:53.801314 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:53.801281 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gghzs" event={"ID":"74c1a604-b97f-4033-919d-6af5f3c9993c","Type":"ContainerStarted","Data":"66af5c0c10b418849a946dca62366aa6dc04f0596cd2c403a6c190757ed20ad3"} Jul 16 13:41:54.407922 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:54.407867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:54.508397 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:54.508363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:55.848786 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:55.848727 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:55.945089 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:55.945013 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.007221 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.007185 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.149159 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.149070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.546859 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.546826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.617011 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.616975 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.715199 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.715165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.770268 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.770235 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:56.924698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:56.924600 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:57.327866 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.327820 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:57.443838 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.443785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:57.542548 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.542512 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:57.830169 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.830127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:57.830331 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.830176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:57.832519 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.832496 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3488b5d2-1e52-4605-91b3-7a0248cffd18-metrics-tls\") pod \"dns-default-rrd8j\" (UID: \"3488b5d2-1e52-4605-91b3-7a0248cffd18\") " pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:57.832705 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.832684 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"image-registry-75cc4c6c8f-t5l5z\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:57.839044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.838991 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfg5g\"" Jul 16 13:41:57.846968 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:57.846942 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrd8j" Jul 16 13:41:58.109356 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:58.109269 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xfh2z\"" Jul 16 13:41:58.117506 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:58.117466 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:58.747358 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:58.747325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:58.802020 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:58.801977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:41:59.742598 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.742563 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:41:59.746586 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:59.746548 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998e8a1b_eb66_46e9_a3f6_bf996d21c932.slice/crio-6cde927e1de2ec04eb597c28328109f7a006d9dbfed9f88ce615bf92358bb598 WatchSource:0}: Error finding container 6cde927e1de2ec04eb597c28328109f7a006d9dbfed9f88ce615bf92358bb598: Status 404 returned error can't find the container with id 6cde927e1de2ec04eb597c28328109f7a006d9dbfed9f88ce615bf92358bb598 Jul 16 13:41:59.756980 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.756457 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrd8j"] Jul 16 13:41:59.759746 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:41:59.759715 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3488b5d2_1e52_4605_91b3_7a0248cffd18.slice/crio-a4955082668a70f8e1d112f876858e4d653d1754110834df179d767f0da21d44 WatchSource:0}: Error finding container a4955082668a70f8e1d112f876858e4d653d1754110834df179d767f0da21d44: Status 404 returned error can't find the container with id a4955082668a70f8e1d112f876858e4d653d1754110834df179d767f0da21d44 Jul 16 13:41:59.824606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.824575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" event={"ID":"998e8a1b-eb66-46e9-a3f6-bf996d21c932","Type":"ContainerStarted","Data":"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578"} Jul 16 13:41:59.824788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.824646 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" event={"ID":"998e8a1b-eb66-46e9-a3f6-bf996d21c932","Type":"ContainerStarted","Data":"6cde927e1de2ec04eb597c28328109f7a006d9dbfed9f88ce615bf92358bb598"} Jul 16 13:41:59.824788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.824684 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:41:59.826121 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.826096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gghzs" event={"ID":"74c1a604-b97f-4033-919d-6af5f3c9993c","Type":"ContainerStarted","Data":"b12346a9a67f0a58f3864e1a0cdba4ea4eaec95cbbc62b21d10e9844bc439a68"} Jul 16 13:41:59.827118 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.827096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrd8j" event={"ID":"3488b5d2-1e52-4605-91b3-7a0248cffd18","Type":"ContainerStarted","Data":"a4955082668a70f8e1d112f876858e4d653d1754110834df179d767f0da21d44"} Jul 16 13:41:59.844188 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:41:59.844138 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" podStartSLOduration=99.844122226 podStartE2EDuration="1m39.844122226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:41:59.84373852 +0000 UTC m=+99.791855261" watchObservedRunningTime="2026-07-16 13:41:59.844122226 +0000 UTC m=+99.792238942" Jul 16 13:42:00.757573 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.757540 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-t2566"] Jul 16 13:42:00.760744 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.760697 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:00.769056 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.768109 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 13:42:00.770744 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.770719 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6sdd6\"" Jul 16 13:42:00.776771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.776749 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 13:42:00.777803 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.777783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-t2566"] Jul 16 13:42:00.802779 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.802697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:00.852986 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.852948 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqnzr\" (UniqueName: \"kubernetes.io/projected/a40e60d9-fecb-48b2-af40-6d2f4c6002f7-kube-api-access-nqnzr\") pod \"downloads-68c9cbb497-t2566\" (UID: \"a40e60d9-fecb-48b2-af40-6d2f4c6002f7\") " pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:00.893686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.893561 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:00.954200 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.953863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqnzr\" (UniqueName: \"kubernetes.io/projected/a40e60d9-fecb-48b2-af40-6d2f4c6002f7-kube-api-access-nqnzr\") pod \"downloads-68c9cbb497-t2566\" (UID: \"a40e60d9-fecb-48b2-af40-6d2f4c6002f7\") " pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:00.964338 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.964289 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqnzr\" (UniqueName: \"kubernetes.io/projected/a40e60d9-fecb-48b2-af40-6d2f4c6002f7-kube-api-access-nqnzr\") pod \"downloads-68c9cbb497-t2566\" (UID: \"a40e60d9-fecb-48b2-af40-6d2f4c6002f7\") " pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:00.992515 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:00.992459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:01.073324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:01.073285 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:02.735455 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:02.735428 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g8t8z" Jul 16 13:42:03.182608 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.182572 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:03.184825 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.184794 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:03.203767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.203736 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:03.203943 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.203904 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.207291 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.207268 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 13:42:03.207847 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.207825 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4w29z\"" Jul 16 13:42:03.208086 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.208071 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 13:42:03.208178 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.208089 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 13:42:03.208475 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.208456 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 13:42:03.208553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.208523 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 13:42:03.265837 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.265804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:03.274679 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.274843 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwb75\" (UniqueName: \"kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.274843 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274782 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.274843 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.275008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274847 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.275008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.274906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.375626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.375584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwb75\" (UniqueName: \"kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.375817 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.375643 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.375897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.375831 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.375960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.375920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.376018 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.375979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.376069 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.376028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.376605 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.376576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.376732 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.376631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.376732 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.376687 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.378392 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.378370 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.378559 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.378542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.384143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.384095 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwb75\" (UniqueName: \"kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75\") pod \"console-7f6fc7b56b-hlc8h\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:03.515546 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:03.515516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:04.003165 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.003132 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:04.006226 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:04.006199 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250f34cb_7914_4e8f_a5f3_764bcc461ff7.slice/crio-54182bed1c250619394457d20a50b8f2dd34a35987c397175ef7f184c5ab26bf WatchSource:0}: Error finding container 54182bed1c250619394457d20a50b8f2dd34a35987c397175ef7f184c5ab26bf: Status 404 returned error can't find the container with id 54182bed1c250619394457d20a50b8f2dd34a35987c397175ef7f184c5ab26bf Jul 16 13:42:04.019198 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.019170 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-t2566"] Jul 16 13:42:04.023162 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:04.023137 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40e60d9_fecb_48b2_af40_6d2f4c6002f7.slice/crio-8b96ed5888d8dbde7098fcbdd28e2fd38285ace57e7266eca14119184b11fab7 WatchSource:0}: Error finding container 8b96ed5888d8dbde7098fcbdd28e2fd38285ace57e7266eca14119184b11fab7: Status 404 returned error can't find the container with id 8b96ed5888d8dbde7098fcbdd28e2fd38285ace57e7266eca14119184b11fab7 Jul 16 13:42:04.847442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.847402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrd8j" event={"ID":"3488b5d2-1e52-4605-91b3-7a0248cffd18","Type":"ContainerStarted","Data":"2acd8ce2e95fe56dd76a5e96a88fd51d212770df04ac8b82a06a453073f1687b"} Jul 16 13:42:04.847630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.847474 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rrd8j" Jul 16 13:42:04.847630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.847503 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrd8j" event={"ID":"3488b5d2-1e52-4605-91b3-7a0248cffd18","Type":"ContainerStarted","Data":"1051db231091cd1e03a1cf51b34cda6dbb9fb241e6bccd6c6ab0a99ab6e27f74"} Jul 16 13:42:04.849959 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.849869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gghzs" event={"ID":"74c1a604-b97f-4033-919d-6af5f3c9993c","Type":"ContainerStarted","Data":"8e1f656b7d5d60dbd5442a111299b016e2eba5053a2192b2e61f3e7368034dfe"} Jul 16 13:42:04.851290 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.851260 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f6fc7b56b-hlc8h" event={"ID":"250f34cb-7914-4e8f-a5f3-764bcc461ff7","Type":"ContainerStarted","Data":"54182bed1c250619394457d20a50b8f2dd34a35987c397175ef7f184c5ab26bf"} Jul 16 13:42:04.852633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.852609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-t2566" event={"ID":"a40e60d9-fecb-48b2-af40-6d2f4c6002f7","Type":"ContainerStarted","Data":"8b96ed5888d8dbde7098fcbdd28e2fd38285ace57e7266eca14119184b11fab7"} Jul 16 13:42:04.865086 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:04.864951 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rrd8j" podStartSLOduration=67.756310663 podStartE2EDuration="1m11.864934716s" totalImagesPullingTime="4.108624053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:04.864580493 +0000 UTC m=+104.812697201" watchObservedRunningTime="2026-07-16 13:42:04.864934716 +0000 UTC m=+104.813051423" Jul 16 13:42:05.549975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.549941 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:42:05.558096 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.558066 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:05.565601 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.565578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.565730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.565680 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:42:05.574898 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.574849 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 13:42:05.697679 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhqv\" (UniqueName: \"kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697816 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.698059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.697847 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799294 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799306 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhqv\" (UniqueName: \"kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.799480 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.799456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.800779 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.800306 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.800779 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.800525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.801129 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.801106 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.801197 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.801183 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.804094 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.803867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.805034 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.805011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.809024 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.808982 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhqv\" (UniqueName: \"kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv\") pod \"console-5888675787-zwg2n\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:05.881585 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:05.881550 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:06.075892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:06.075688 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:42:06.863324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:06.863218 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888675787-zwg2n" event={"ID":"135e5c10-bd38-4464-8736-eec057cd9d20","Type":"ContainerStarted","Data":"fb2abb3163f9b2c9b74f118ce69a78cf82465738b94b355ebf8b8778e7ef5040"} Jul 16 13:42:08.575854 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.575763 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:08.678895 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.678800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:08.736075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.736034 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf"] Jul 16 13:42:08.747627 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.747595 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf"] Jul 16 13:42:08.747803 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.747728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:08.750293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.750266 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 13:42:08.750671 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.750649 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qbtnw\"" Jul 16 13:42:08.752311 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.752039 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:08.830152 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.830030 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1766adb1-a0f5-4c59-8792-52d4fb47882e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vldnf\" (UID: \"1766adb1-a0f5-4c59-8792-52d4fb47882e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:08.907440 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.907387 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:08.930841 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.930801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1766adb1-a0f5-4c59-8792-52d4fb47882e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vldnf\" (UID: \"1766adb1-a0f5-4c59-8792-52d4fb47882e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:08.933809 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:08.933782 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1766adb1-a0f5-4c59-8792-52d4fb47882e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vldnf\" (UID: \"1766adb1-a0f5-4c59-8792-52d4fb47882e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:09.060210 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.060174 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:09.704117 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.704070 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf"] Jul 16 13:42:09.708137 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:09.708105 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1766adb1_a0f5_4c59_8792_52d4fb47882e.slice/crio-a061f900f0aa608381b22a6b4b2e6be5de1e77bd848fc679c2a60fdd4fe69ee9 WatchSource:0}: Error finding container a061f900f0aa608381b22a6b4b2e6be5de1e77bd848fc679c2a60fdd4fe69ee9: Status 404 returned error can't find the container with id a061f900f0aa608381b22a6b4b2e6be5de1e77bd848fc679c2a60fdd4fe69ee9 Jul 16 13:42:09.875151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.875046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" event={"ID":"1766adb1-a0f5-4c59-8792-52d4fb47882e","Type":"ContainerStarted","Data":"a061f900f0aa608381b22a6b4b2e6be5de1e77bd848fc679c2a60fdd4fe69ee9"} Jul 16 13:42:09.877431 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.877326 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gghzs" event={"ID":"74c1a604-b97f-4033-919d-6af5f3c9993c","Type":"ContainerStarted","Data":"2c323f1551cae8690f36427897543a24bdbdcdcf65905b302692cc7fb3abd5e6"} Jul 16 13:42:09.879131 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.879108 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f6fc7b56b-hlc8h" event={"ID":"250f34cb-7914-4e8f-a5f3-764bcc461ff7","Type":"ContainerStarted","Data":"d1185604d897610aa42a604be6c95100af13a624e122bf234297b3b802dc20d2"} Jul 16 13:42:09.880831 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.880808 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888675787-zwg2n" event={"ID":"135e5c10-bd38-4464-8736-eec057cd9d20","Type":"ContainerStarted","Data":"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d"} Jul 16 13:42:09.896920 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.896843 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gghzs" podStartSLOduration=16.916942223 podStartE2EDuration="32.896823939s" totalImagesPullingTime="15.979881716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:41:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:09.895347011 +0000 UTC m=+109.843463740" watchObservedRunningTime="2026-07-16 13:42:09.896823939 +0000 UTC m=+109.844940646" Jul 16 13:42:09.911442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.911393 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5888675787-zwg2n" podStartSLOduration=1.4585715430000001 podStartE2EDuration="4.911374245s" totalImagesPullingTime="3.452802702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:09.910953335 +0000 UTC m=+109.859070062" watchObservedRunningTime="2026-07-16 13:42:09.911374245 +0000 UTC m=+109.859490932" Jul 16 13:42:09.928318 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:09.928263 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f6fc7b56b-hlc8h" podStartSLOduration=1.401043297 podStartE2EDuration="6.928242499s" totalImagesPullingTime="5.527199202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:09.927486628 +0000 UTC m=+109.875603350" watchObservedRunningTime="2026-07-16 13:42:09.928242499 +0000 UTC m=+109.876359208" Jul 16 13:42:10.886623 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:10.886579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" event={"ID":"1766adb1-a0f5-4c59-8792-52d4fb47882e","Type":"ContainerStarted","Data":"553add7f5b20e7458ee662d984ce86b4f71d9057ef2f43b4eaf046bb7b4aa61d"} Jul 16 13:42:10.887111 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:10.886960 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:10.888475 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:10.888448 2563 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-77d98bc786-vldnf container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/healthz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Jul 16 13:42:10.888573 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:10.888498 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" podUID="1766adb1-a0f5-4c59-8792-52d4fb47882e" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.15:8443/healthz\": dial tcp 10.132.0.15:8443: connect: connection refused" Jul 16 13:42:10.901743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:10.901686 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" podStartSLOduration=1.806125641 podStartE2EDuration="2.901667412s" totalImagesPullingTime="1.095541771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:10.900862976 +0000 UTC m=+110.848979674" watchObservedRunningTime="2026-07-16 13:42:10.901667412 +0000 UTC m=+110.849784100" Jul 16 13:42:11.151823 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:11.151734 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:11.895142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:11.895114 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vldnf" Jul 16 13:42:12.771643 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.771601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:12.808839 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.808810 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-2xgrn"] Jul 16 13:42:12.812742 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.812718 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:12.816562 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.816544 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 13:42:12.816668 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.816634 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 13:42:12.817311 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.817290 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9592d\"" Jul 16 13:42:12.817543 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.817457 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 13:42:12.817543 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.817483 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 13:42:12.817771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.817554 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 13:42:12.822232 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.822205 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-2xgrn"] Jul 16 13:42:12.845074 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.845042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:12.965642 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.965599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzgq\" (UniqueName: \"kubernetes.io/projected/b84787a6-342d-4c14-a474-2b79d9958ff9-kube-api-access-2vzgq\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:12.965642 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.965659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b84787a6-342d-4c14-a474-2b79d9958ff9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:12.966128 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.965780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:12.966128 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:12.965821 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.018685 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.018643 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:13.066797 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.066704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzgq\" (UniqueName: \"kubernetes.io/projected/b84787a6-342d-4c14-a474-2b79d9958ff9-kube-api-access-2vzgq\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.066990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.066765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b84787a6-342d-4c14-a474-2b79d9958ff9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.066990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.066894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.066990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.066955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.067612 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.067587 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b84787a6-342d-4c14-a474-2b79d9958ff9-metrics-client-ca\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.069613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.069588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.069827 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.069808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b84787a6-342d-4c14-a474-2b79d9958ff9-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.076491 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.076442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzgq\" (UniqueName: \"kubernetes.io/projected/b84787a6-342d-4c14-a474-2b79d9958ff9-kube-api-access-2vzgq\") pod \"prometheus-operator-64785b6675-2xgrn\" (UID: \"b84787a6-342d-4c14-a474-2b79d9958ff9\") " pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.124030 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.123995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" Jul 16 13:42:13.271184 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.270937 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-2xgrn"] Jul 16 13:42:13.274136 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:13.274096 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84787a6_342d_4c14_a474_2b79d9958ff9.slice/crio-7e98649ee50f453a773da3bb5ccd10174d1c556a753cb2e7615bdffcf1948014 WatchSource:0}: Error finding container 7e98649ee50f453a773da3bb5ccd10174d1c556a753cb2e7615bdffcf1948014: Status 404 returned error can't find the container with id 7e98649ee50f453a773da3bb5ccd10174d1c556a753cb2e7615bdffcf1948014 Jul 16 13:42:13.516163 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.516126 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:13.516333 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.516195 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:13.517746 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.517724 2563 patch_prober.go:28] interesting pod/console-7f6fc7b56b-hlc8h container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.13:8443/health\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Jul 16 13:42:13.517845 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.517775 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7f6fc7b56b-hlc8h" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerName="console" probeResult="failure" output="Get \"https://10.132.0.13:8443/health\": dial tcp 10.132.0.13:8443: connect: connection refused" Jul 16 13:42:13.904820 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:13.904738 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" event={"ID":"b84787a6-342d-4c14-a474-2b79d9958ff9","Type":"ContainerStarted","Data":"7e98649ee50f453a773da3bb5ccd10174d1c556a753cb2e7615bdffcf1948014"} Jul 16 13:42:14.860798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:14.860762 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rrd8j" Jul 16 13:42:15.166433 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:15.166338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:15.882529 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:15.882489 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:15.882983 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:15.882555 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:15.884288 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:15.884262 2563 patch_prober.go:28] interesting pod/console-5888675787-zwg2n container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Jul 16 13:42:15.884404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:15.884312 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5888675787-zwg2n" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" containerName="console" probeResult="failure" output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused" Jul 16 13:42:18.123835 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:18.123794 2563 patch_prober.go:28] interesting pod/image-registry-75cc4c6c8f-t5l5z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 13:42:18.124259 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:18.123848 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 13:42:19.223531 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:19.223492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:19.324264 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:19.324221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:20.836604 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:20.836575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:42:21.007888 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:21.007846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:21.098791 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:21.098680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:21.174555 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:21.174513 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:22.299319 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.299262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:22.942132 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.942090 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" event={"ID":"b84787a6-342d-4c14-a474-2b79d9958ff9","Type":"ContainerStarted","Data":"8052d5272ec2536d96f711b27861020414e189245c577d71ed977ec15f00fde7"} Jul 16 13:42:22.942338 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.942168 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" event={"ID":"b84787a6-342d-4c14-a474-2b79d9958ff9","Type":"ContainerStarted","Data":"8c0ec5e12a7d7ea7ed2ba6b496e0b3967717e4414f9e6677c2b9df12872e07a3"} Jul 16 13:42:22.943980 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.943947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-t2566" event={"ID":"a40e60d9-fecb-48b2-af40-6d2f4c6002f7","Type":"ContainerStarted","Data":"9c884235343072668420cfb8799254e3fd9850807f915741908fd6b4227bd4cc"} Jul 16 13:42:22.944190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.944167 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:22.953953 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.953929 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-t2566" Jul 16 13:42:22.960752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.960704 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-2xgrn" podStartSLOduration=2.288746817 podStartE2EDuration="10.960686601s" totalImagesPullingTime="8.671939784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:22.960208042 +0000 UTC m=+122.908324748" watchObservedRunningTime="2026-07-16 13:42:22.960686601 +0000 UTC m=+122.908803305" Jul 16 13:42:22.977042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:22.976988 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-t2566" podStartSLOduration=5.009609675 podStartE2EDuration="22.976973746s" totalImagesPullingTime="17.967364071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:22.975860813 +0000 UTC m=+122.923977541" watchObservedRunningTime="2026-07-16 13:42:22.976973746 +0000 UTC m=+122.925090452" Jul 16 13:42:23.362410 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.362376 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:42:23.458067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.458029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:23.521612 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.521582 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:23.526143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.526116 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:23.543034 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.542999 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:23.615009 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.614910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:23.715146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:23.715114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:24.026072 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:24.026034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:24.459685 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:24.459591 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:24.782929 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:24.782891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:24.888840 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:24.888806 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:24.994298 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:24.994262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.177378 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.177292 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.203477 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.203438 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf"] Jul 16 13:42:25.228164 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.228134 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cmjmt"] Jul 16 13:42:25.228500 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.228474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.239236 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.239211 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tpznr\"" Jul 16 13:42:25.239699 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.239681 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 13:42:25.240066 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.240050 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 13:42:25.251386 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.251349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf"] Jul 16 13:42:25.251508 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.251461 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.254146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.254099 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89v2l\"" Jul 16 13:42:25.254301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.254243 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 13:42:25.254301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.254239 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 13:42:25.254587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.254569 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 13:42:25.274067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274039 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgv9\" (UniqueName: \"kubernetes.io/projected/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-kube-api-access-qzgv9\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.274222 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pl45\" (UniqueName: \"kubernetes.io/projected/060ead13-48f4-4343-881f-85fbc2575782-kube-api-access-2pl45\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274298 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-metrics-client-ca\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274355 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.274413 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274395 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.274475 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.274528 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-root\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274528 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-textfile\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274528 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274645 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274539 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-sys\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274645 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274555 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274645 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-wtmp\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.274775 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.274676 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.321735 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.321691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.375891 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-root\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-textfile\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375943 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-root\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-sys\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.375978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-wtmp\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376036 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-sys\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376075 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:42:25.376064 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:42:25.376127 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls podName:060ead13-48f4-4343-881f-85fbc2575782 nodeName:}" failed. No retries permitted until 2026-07-16 13:42:25.876106461 +0000 UTC m=+125.824223148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls") pod "node-exporter-cmjmt" (UID: "060ead13-48f4-4343-881f-85fbc2575782") : secret "node-exporter-tls" not found Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376229 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-wtmp\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgv9\" (UniqueName: \"kubernetes.io/projected/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-kube-api-access-qzgv9\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376296 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-textfile\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pl45\" (UniqueName: \"kubernetes.io/projected/060ead13-48f4-4343-881f-85fbc2575782-kube-api-access-2pl45\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-metrics-client-ca\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376399 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376461 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.376535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.377020 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376722 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.377020 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.376851 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060ead13-48f4-4343-881f-85fbc2575782-metrics-client-ca\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.377290 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.377262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.378984 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.378959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.379251 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.379221 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.379360 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.379342 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.387315 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.387262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pl45\" (UniqueName: \"kubernetes.io/projected/060ead13-48f4-4343-881f-85fbc2575782-kube-api-access-2pl45\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.388265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.388242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgv9\" (UniqueName: \"kubernetes.io/projected/4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e-kube-api-access-qzgv9\") pod \"openshift-state-metrics-84bd68b896-pvlqf\" (UID: \"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.394254 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.394225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.495126 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.495040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.540891 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.540834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" Jul 16 13:42:25.710574 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.710516 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.719854 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.719817 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf"] Jul 16 13:42:25.722949 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:25.722920 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8e4397_f7e1_4fd4_a9cf_6cd6f7d1b42e.slice/crio-2f2ffc2a342759f1ab572ef9c6b6d3c9f63f43c5f322d20da2d447a8e9805c41 WatchSource:0}: Error finding container 2f2ffc2a342759f1ab572ef9c6b6d3c9f63f43c5f322d20da2d447a8e9805c41: Status 404 returned error can't find the container with id 2f2ffc2a342759f1ab572ef9c6b6d3c9f63f43c5f322d20da2d447a8e9805c41 Jul 16 13:42:25.808494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.808458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:25.881075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.881032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.883932 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.883901 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/060ead13-48f4-4343-881f-85fbc2575782-node-exporter-tls\") pod \"node-exporter-cmjmt\" (UID: \"060ead13-48f4-4343-881f-85fbc2575782\") " pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:25.888078 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.888060 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:25.892615 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.892593 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:42:25.958250 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.958219 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:25.959660 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.959589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" event={"ID":"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e","Type":"ContainerStarted","Data":"120bd62dfb473d0a81e9b18464d6f0c351e137c06477d5413a212a2ba2404a82"} Jul 16 13:42:25.959660 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:25.959658 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" event={"ID":"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e","Type":"ContainerStarted","Data":"2f2ffc2a342759f1ab572ef9c6b6d3c9f63f43c5f322d20da2d447a8e9805c41"} Jul 16 13:42:26.078580 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.078528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:26.162361 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.162323 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmjmt" Jul 16 13:42:26.219764 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.219725 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:26.320703 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.320671 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 13:42:26.328007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.327979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:26.344682 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.344595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.348746 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.348714 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 13:42:26.349158 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.349125 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 13:42:26.349270 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.349170 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 13:42:26.349421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.349389 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 13:42:26.349497 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.349474 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 13:42:26.350099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.350039 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 13:42:26.350213 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.350153 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 13:42:26.350213 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.350049 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-74lh8\"" Jul 16 13:42:26.350382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.350359 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 13:42:26.350595 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.350572 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 13:42:26.352420 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.352398 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 13:42:26.389644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-web-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389719 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389744 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389867 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.389975 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.389927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlf7\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-kube-api-access-xhlf7\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.390278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.390014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.390278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.390114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.390278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.390188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-config-out\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.390278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.390263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-config-volume\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.444772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.444727 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:26.490751 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490768 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-config-out\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-config-volume\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-web-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490895 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.490990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.490921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491411 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlf7\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-kube-api-access-xhlf7\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.491614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.492314 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.492802 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.492781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.494477 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.494379 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.494985 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.494937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.495108 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.495074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.495334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.495311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6073e961-6fde-430b-9e8f-9db36e67f622-config-out\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.495696 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.495562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-config-volume\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.495696 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.495602 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.496118 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.496089 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.496412 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.496386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073e961-6fde-430b-9e8f-9db36e67f622-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.496631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.496608 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.497287 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.497269 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6073e961-6fde-430b-9e8f-9db36e67f622-web-config\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.501167 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.501139 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlf7\" (UniqueName: \"kubernetes.io/projected/6073e961-6fde-430b-9e8f-9db36e67f622-kube-api-access-xhlf7\") pod \"alertmanager-main-0\" (UID: \"6073e961-6fde-430b-9e8f-9db36e67f622\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.658224 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.658140 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 13:42:26.781505 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.781465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:26.867988 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.867951 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 13:42:26.967549 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.967409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" event={"ID":"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e","Type":"ContainerStarted","Data":"710d8a6fdf5d9c42a1545a5af650ba1fdaabf02286a4e0ea6664debcac94aac8"} Jul 16 13:42:26.969217 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.969134 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"2a76d73942794e745ad38f311caca966a1c084fb900f1eec85e79087a23cdaf3"} Jul 16 13:42:26.971018 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:26.970895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmjmt" event={"ID":"060ead13-48f4-4343-881f-85fbc2575782","Type":"ContainerStarted","Data":"05aa91e3dfecab1b1d00d22c2d1fb074a24cb9856164780f16734a74d7adc389"} Jul 16 13:42:27.218810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.218706 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-557855ff85-l5b4d"] Jul 16 13:42:27.247007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.246974 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-557855ff85-l5b4d"] Jul 16 13:42:27.247215 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.247153 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.250156 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.250113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 13:42:27.251151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.250907 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 13:42:27.252051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.251302 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 13:42:27.252051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.251351 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 13:42:27.252051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.251556 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-znv6g\"" Jul 16 13:42:27.252051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.251650 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1aa8b6bak784t\"" Jul 16 13:42:27.252051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.251909 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 13:42:27.298073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7774c354-37dd-41b2-8ece-bcbd47bd7c56-metrics-client-ca\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298345 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfcrs\" (UniqueName: \"kubernetes.io/projected/7774c354-37dd-41b2-8ece-bcbd47bd7c56-kube-api-access-dfcrs\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.298457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.298445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-grpc-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.399791 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.399758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400002 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.399813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400167 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7774c354-37dd-41b2-8ece-bcbd47bd7c56-metrics-client-ca\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400420 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcrs\" (UniqueName: \"kubernetes.io/projected/7774c354-37dd-41b2-8ece-bcbd47bd7c56-kube-api-access-dfcrs\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400420 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.400420 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.400388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-grpc-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.401237 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.401207 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7774c354-37dd-41b2-8ece-bcbd47bd7c56-metrics-client-ca\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.404552 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.404491 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.405661 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.405600 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-grpc-tls\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.406170 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.406114 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.406703 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.406386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.406703 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.406657 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.407299 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.407270 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7774c354-37dd-41b2-8ece-bcbd47bd7c56-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.410404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.410384 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcrs\" (UniqueName: \"kubernetes.io/projected/7774c354-37dd-41b2-8ece-bcbd47bd7c56-kube-api-access-dfcrs\") pod \"thanos-querier-557855ff85-l5b4d\" (UID: \"7774c354-37dd-41b2-8ece-bcbd47bd7c56\") " pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:27.561238 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:27.561202 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:28.190610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.190576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:28.204897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.204776 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-557855ff85-l5b4d"] Jul 16 13:42:28.211622 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:28.211587 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7774c354_37dd_41b2_8ece_bcbd47bd7c56.slice/crio-53f9b4a4da8d2a44afd908aa129a614ea0ea3905b2ba386699d557904645cff9 WatchSource:0}: Error finding container 53f9b4a4da8d2a44afd908aa129a614ea0ea3905b2ba386699d557904645cff9: Status 404 returned error can't find the container with id 53f9b4a4da8d2a44afd908aa129a614ea0ea3905b2ba386699d557904645cff9 Jul 16 13:42:28.283859 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.283828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:28.797172 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.797139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:28.901930 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.901895 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:28.982613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.982574 2563 generic.go:356] "Generic (PLEG): container finished" podID="060ead13-48f4-4343-881f-85fbc2575782" containerID="68cab5655abce8e6fe2c0e500f972893f3f16846725d5e12adb01183c176538a" exitCode=0 Jul 16 13:42:28.982789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.982699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmjmt" event={"ID":"060ead13-48f4-4343-881f-85fbc2575782","Type":"ContainerDied","Data":"68cab5655abce8e6fe2c0e500f972893f3f16846725d5e12adb01183c176538a"} Jul 16 13:42:28.986229 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.986200 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" event={"ID":"4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e","Type":"ContainerStarted","Data":"3e2aab7e38177825f9b8fa2f5e8500d925b42dbee1edc3057cbda064ff2a5de4"} Jul 16 13:42:28.987690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:28.987624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"53f9b4a4da8d2a44afd908aa129a614ea0ea3905b2ba386699d557904645cff9"} Jul 16 13:42:29.027231 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.027185 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pvlqf" podStartSLOduration=2.00935012 podStartE2EDuration="4.027168664s" totalImagesPullingTime="2.017818544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:29.025294296 +0000 UTC m=+128.973411029" watchObservedRunningTime="2026-07-16 13:42:29.027168664 +0000 UTC m=+128.975285371" Jul 16 13:42:29.319653 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.319582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:42:29.323691 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.323636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c18e82a-f845-4475-bc78-f29acf434c83-metrics-certs\") pod \"network-metrics-daemon-kf4n6\" (UID: \"8c18e82a-f845-4475-bc78-f29acf434c83\") " pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:42:29.385246 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.385107 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6cxrw\"" Jul 16 13:42:29.391635 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.391438 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kf4n6" Jul 16 13:42:29.659074 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.658947 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kf4n6"] Jul 16 13:42:29.696156 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:29.695081 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c18e82a_f845_4475_bc78_f29acf434c83.slice/crio-e17c47d7e44c2469ac1b6014743af6e538df0615f23754963ac7649a2bbb71e2 WatchSource:0}: Error finding container e17c47d7e44c2469ac1b6014743af6e538df0615f23754963ac7649a2bbb71e2: Status 404 returned error can't find the container with id e17c47d7e44c2469ac1b6014743af6e538df0615f23754963ac7649a2bbb71e2 Jul 16 13:42:29.968496 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.968410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:29.998470 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.997897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmjmt" event={"ID":"060ead13-48f4-4343-881f-85fbc2575782","Type":"ContainerStarted","Data":"2df32416678baebe9d9e7e38d775c891f43b0716317850d6e26ffca4aac0c245"} Jul 16 13:42:29.998470 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.997965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmjmt" event={"ID":"060ead13-48f4-4343-881f-85fbc2575782","Type":"ContainerStarted","Data":"8194fa267ca567867ab33b3dbb0fc8602fbb3da2b62f7e619b3ff72004b8f70f"} Jul 16 13:42:29.999808 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:29.999778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kf4n6" event={"ID":"8c18e82a-f845-4475-bc78-f29acf434c83","Type":"ContainerStarted","Data":"e17c47d7e44c2469ac1b6014743af6e538df0615f23754963ac7649a2bbb71e2"} Jul 16 13:42:30.001972 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.001940 2563 generic.go:356] "Generic (PLEG): container finished" podID="6073e961-6fde-430b-9e8f-9db36e67f622" containerID="2e8b43487e16e7538cf0f91836ef03176d587f97bbc469a5f5aac2f8c4ce026f" exitCode=0 Jul 16 13:42:30.002522 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.002494 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerDied","Data":"2e8b43487e16e7538cf0f91836ef03176d587f97bbc469a5f5aac2f8c4ce026f"} Jul 16 13:42:30.024076 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.023280 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cmjmt" podStartSLOduration=3.115947829 podStartE2EDuration="5.023258582s" totalImagesPullingTime="1.821022394s" totalInitContainerRuntime="86.288359ms" isStatefulPod=true podCreationTimestamp="2026-07-16 13:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:30.022367875 +0000 UTC m=+129.970484607" watchObservedRunningTime="2026-07-16 13:42:30.023258582 +0000 UTC m=+129.971375291" Jul 16 13:42:30.072069 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.072025 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:30.178656 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.178577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:30.559134 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.559084 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:42:30.595536 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.595502 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:42:30.595793 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.595631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.632761 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.632705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.632761 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.632763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.633120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.632800 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.633120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.632837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgjh\" (UniqueName: \"kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.633120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.632934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.633120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.633022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.633120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.633050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.734095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.734050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.734917 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.734824 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.734917 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.734899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.734950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.734988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735150 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.735137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735314 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.735196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgjh\" (UniqueName: \"kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735314 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.735244 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.735564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.735502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.736553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.736525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.737249 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.737169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.739527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.739480 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.740356 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.740332 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.743942 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.743914 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgjh\" (UniqueName: \"kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh\") pod \"console-5d5c5dd89d-w6t4r\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:30.780746 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.780686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:30.841535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.841459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:30.909143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:30.909099 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:31.087341 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:31.087309 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:42:31.119214 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:31.119142 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b206cc_51b6_4d9e_bf69_a9a0af37001b.slice/crio-014f4d6f98e0b7f284b04a885dd1de11bc6437494f2b946642cb6aaf2a72d1b1 WatchSource:0}: Error finding container 014f4d6f98e0b7f284b04a885dd1de11bc6437494f2b946642cb6aaf2a72d1b1: Status 404 returned error can't find the container with id 014f4d6f98e0b7f284b04a885dd1de11bc6437494f2b946642cb6aaf2a72d1b1 Jul 16 13:42:31.548526 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:31.548449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:31.668769 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:31.668649 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:31.770051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:31.769830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:31.869852 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:31.869816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:32.015119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.014917 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"2edc5dc3460a51a08aa52471c8fa141f3c0bcae2fcfc46db6ed4731b48670e22"} Jul 16 13:42:32.015119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.015012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"52fcdca21380db363ee92bbf498df47321dc8d24ab3b9adb7e41b278bb3e13ad"} Jul 16 13:42:32.015119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.015054 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"0ebcdfb0d1a0c649897e41da6c0c3a012169c384de9de1871f45974e6fbd8479"} Jul 16 13:42:32.017428 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.017148 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c5dd89d-w6t4r" event={"ID":"12b206cc-51b6-4d9e-bf69-a9a0af37001b","Type":"ContainerStarted","Data":"74b139f8636c8fd58bd5764998794f3a7eab5c6375ce6c7cdb830c64cacbe008"} Jul 16 13:42:32.017428 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.017194 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c5dd89d-w6t4r" event={"ID":"12b206cc-51b6-4d9e-bf69-a9a0af37001b","Type":"ContainerStarted","Data":"014f4d6f98e0b7f284b04a885dd1de11bc6437494f2b946642cb6aaf2a72d1b1"} Jul 16 13:42:32.035860 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:32.035812 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5c5dd89d-w6t4r" podStartSLOduration=2.035792235 podStartE2EDuration="2.035792235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:32.034464521 +0000 UTC m=+131.982581229" watchObservedRunningTime="2026-07-16 13:42:32.035792235 +0000 UTC m=+131.983908983" Jul 16 13:42:34.030160 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.030100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"3fb7bdc3965cd2dd6c65ad57d186e6b59083572849028fe48b06a1211d5edc16"} Jul 16 13:42:34.031219 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.030221 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"616e10e971e67343f925e6f449976776f0427d2b47f884d13b39251ecc851412"} Jul 16 13:42:34.031219 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.030268 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" event={"ID":"7774c354-37dd-41b2-8ece-bcbd47bd7c56","Type":"ContainerStarted","Data":"316ec825346228a710bdbb48402dc8148924b9f354d84fa000d7bd1528c863b2"} Jul 16 13:42:34.031219 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.030439 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:34.034261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034236 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"f60c246c0385105eeebe46038e6f56b05ad21d22558a2a4d45fd5305b588974b"} Jul 16 13:42:34.034381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"795151f529f78e2a3defd32e156cd25c4188c0a94c25223ab91f1d2b8842f02b"} Jul 16 13:42:34.034381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034332 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"634cc6f99410c2a4b83e6c18f286cfeb1e4a3e33882849b901d4e0111ae8c48f"} Jul 16 13:42:34.034381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"b3aec80c4a480a3625248002287db55e358bae0c9dce55a06ab4197e516a89d5"} Jul 16 13:42:34.034493 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"9c338d33f4d6e6446210314e537a06d1d743fa56604bfd7c535276cc5bf9826e"} Jul 16 13:42:34.034493 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.034444 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6073e961-6fde-430b-9e8f-9db36e67f622","Type":"ContainerStarted","Data":"07204081a22fdd6d259cc236a3683d4f1d95f506b63131dd04ed1aca93b1d813"} Jul 16 13:42:34.036056 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.036030 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kf4n6" event={"ID":"8c18e82a-f845-4475-bc78-f29acf434c83","Type":"ContainerStarted","Data":"1d0237d827c07b08bd409a2d2f602ae749d32ebba93e4244455928a12e570eae"} Jul 16 13:42:34.036146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.036081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kf4n6" event={"ID":"8c18e82a-f845-4475-bc78-f29acf434c83","Type":"ContainerStarted","Data":"09941604aee5a2d584bd4ec5c2939074c21979525e11c3c1a8b15e9df8549891"} Jul 16 13:42:34.057794 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.057743 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" podStartSLOduration=2.475895228 podStartE2EDuration="7.057726802s" totalImagesPullingTime="4.581831574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:34.053958759 +0000 UTC m=+134.002075487" watchObservedRunningTime="2026-07-16 13:42:34.057726802 +0000 UTC m=+134.005843508" Jul 16 13:42:34.083753 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.083701 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.282074743 podStartE2EDuration="8.083682324s" totalImagesPullingTime="5.7786258s" totalInitContainerRuntime="22.981781ms" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:34.081843102 +0000 UTC m=+134.029959819" watchObservedRunningTime="2026-07-16 13:42:34.083682324 +0000 UTC m=+134.031799031" Jul 16 13:42:34.100638 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.100567 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kf4n6" podStartSLOduration=130.627505694 podStartE2EDuration="2m14.100551826s" totalImagesPullingTime="3.473046132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:34.099042314 +0000 UTC m=+134.047159022" watchObservedRunningTime="2026-07-16 13:42:34.100551826 +0000 UTC m=+134.048668534" Jul 16 13:42:34.200364 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.200324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:34.265296 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.265257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:34.833416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.833381 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:34.925940 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.925905 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:42:34.928923 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.928889 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:34.961809 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.961780 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:42:34.994331 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.994296 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:42:34.994486 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:34.994391 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.081397 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.081352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.081898 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.081409 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.081898 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.081652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.081898 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.081701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4s2\" (UniqueName: \"kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.082067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.081939 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.082250 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.082227 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.082376 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.082356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.121852 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.121749 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:35.183250 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183408 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183408 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183483 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183483 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183463 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183556 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183488 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.183556 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.183503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4s2\" (UniqueName: \"kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.184111 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.184083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.184255 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.184238 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.184323 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.184266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.184323 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.184273 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.185673 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.185654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.185888 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.185856 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.187824 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.187801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:35.190867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.190847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4s2\" (UniqueName: \"kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2\") pod \"console-7dbcf88664-q5qhm\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.305207 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.305171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:35.481051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.481019 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:42:35.482529 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:42:35.482507 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04beb06_dd59_45c4_b40a_4fee126a6566.slice/crio-7beaf8bb075510389212fd6755097b787cdc5dbce75d0548753a3a9d5fb86b97 WatchSource:0}: Error finding container 7beaf8bb075510389212fd6755097b787cdc5dbce75d0548753a3a9d5fb86b97: Status 404 returned error can't find the container with id 7beaf8bb075510389212fd6755097b787cdc5dbce75d0548753a3a9d5fb86b97 Jul 16 13:42:35.819763 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:35.819729 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:36.046769 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:36.046737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf88664-q5qhm" event={"ID":"e04beb06-dd59-45c4-b40a-4fee126a6566","Type":"ContainerStarted","Data":"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2"} Jul 16 13:42:36.046951 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:36.046791 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf88664-q5qhm" event={"ID":"e04beb06-dd59-45c4-b40a-4fee126a6566","Type":"ContainerStarted","Data":"7beaf8bb075510389212fd6755097b787cdc5dbce75d0548753a3a9d5fb86b97"} Jul 16 13:42:36.066323 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:36.066276 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dbcf88664-q5qhm" podStartSLOduration=2.06625873 podStartE2EDuration="2.06625873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:42:36.064470806 +0000 UTC m=+136.012587523" watchObservedRunningTime="2026-07-16 13:42:36.06625873 +0000 UTC m=+136.014375515" Jul 16 13:42:37.165126 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:37.165085 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:39.426085 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:39.426056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:39.522598 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:39.522556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:40.047474 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:40.047449 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-557855ff85-l5b4d" Jul 16 13:42:40.909772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:40.909734 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:42:43.835562 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:43.835525 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:43.938129 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:43.938087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:45.306179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:45.306135 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:45.306637 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:45.306202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:45.312426 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:45.312402 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:46.091258 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:46.091230 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:42:46.141098 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:46.141060 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:42:46.251469 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:46.251429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:46.354720 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:46.354625 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:46.422886 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:46.422849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:47.277271 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:47.277237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:47.347808 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:47.347774 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:48.391818 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.391784 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerName="registry" containerID="cri-o://1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578" gracePeriod=30 Jul 16 13:42:48.614182 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.614142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:48.679666 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.679636 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:42:48.693552 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.693523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:48.803988 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.803956 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804014 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804042 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlchf\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804077 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804100 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804129 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804166 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804161 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804455 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804189 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates\") pod \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\" (UID: \"998e8a1b-eb66-46e9-a3f6-bf996d21c932\") " Jul 16 13:42:48.804509 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804469 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:42:48.804789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.804754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:42:48.806721 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.806689 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:42:48.806864 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.806843 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:42:48.807160 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.807132 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:42:48.807263 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.807165 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:42:48.807263 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.807229 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf" (OuterVolumeSpecName: "kube-api-access-hlchf") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "kube-api-access-hlchf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:42:48.813556 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.813531 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "998e8a1b-eb66-46e9-a3f6-bf996d21c932" (UID: "998e8a1b-eb66-46e9-a3f6-bf996d21c932"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.904952 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-trusted-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.904984 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-image-registry-private-configuration\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.904997 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlchf\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-kube-api-access-hlchf\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.905007 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-tls\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.905015 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/998e8a1b-eb66-46e9-a3f6-bf996d21c932-bound-sa-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.905023 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/998e8a1b-eb66-46e9-a3f6-bf996d21c932-ca-trust-extracted\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.905031 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/998e8a1b-eb66-46e9-a3f6-bf996d21c932-installation-pull-secrets\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:48.905041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:48.905038 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/998e8a1b-eb66-46e9-a3f6-bf996d21c932-registry-certificates\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:49.100027 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.099995 2563 generic.go:356] "Generic (PLEG): container finished" podID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerID="1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578" exitCode=0 Jul 16 13:42:49.100226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.100039 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" Jul 16 13:42:49.100226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.100078 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" event={"ID":"998e8a1b-eb66-46e9-a3f6-bf996d21c932","Type":"ContainerDied","Data":"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578"} Jul 16 13:42:49.100226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.100133 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75cc4c6c8f-t5l5z" event={"ID":"998e8a1b-eb66-46e9-a3f6-bf996d21c932","Type":"ContainerDied","Data":"6cde927e1de2ec04eb597c28328109f7a006d9dbfed9f88ce615bf92358bb598"} Jul 16 13:42:49.100226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.100173 2563 scope.go:117] "RemoveContainer" containerID="1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578" Jul 16 13:42:49.109119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.109102 2563 scope.go:117] "RemoveContainer" containerID="1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578" Jul 16 13:42:49.109367 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:42:49.109349 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578\": container with ID starting with 1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578 not found: ID does not exist" containerID="1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578" Jul 16 13:42:49.109438 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.109376 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578"} err="failed to get container status \"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578\": rpc error: code = NotFound desc = could not find container \"1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578\": container with ID starting with 1b48831e06a440b82cbc9ba01e8778e4c0abee871500fca6d2909fca91bd1578 not found: ID does not exist" Jul 16 13:42:49.133310 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.133277 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:42:49.146613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:49.146589 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75cc4c6c8f-t5l5z"] Jul 16 13:42:50.473325 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:50.473294 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" path="/var/lib/kubelet/pods/998e8a1b-eb66-46e9-a3f6-bf996d21c932/volumes" Jul 16 13:42:50.986065 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:50.986030 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f6fc7b56b-hlc8h" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerName="console" containerID="cri-o://d1185604d897610aa42a604be6c95100af13a624e122bf234297b3b802dc20d2" gracePeriod=15 Jul 16 13:42:51.110230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.110198 2563 generic.go:356] "Generic (PLEG): container finished" podID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerID="d1185604d897610aa42a604be6c95100af13a624e122bf234297b3b802dc20d2" exitCode=0 Jul 16 13:42:51.110392 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.110267 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f6fc7b56b-hlc8h" event={"ID":"250f34cb-7914-4e8f-a5f3-764bcc461ff7","Type":"ContainerDied","Data":"d1185604d897610aa42a604be6c95100af13a624e122bf234297b3b802dc20d2"} Jul 16 13:42:51.270230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.270198 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:51.428929 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.428868 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.428964 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.428999 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429043 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429067 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429088 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwb75\" (UniqueName: \"kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75\") pod \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\" (UID: \"250f34cb-7914-4e8f-a5f3-764bcc461ff7\") " Jul 16 13:42:51.429441 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429398 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:42:51.429586 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429413 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca" (OuterVolumeSpecName: "service-ca") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:42:51.429586 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.429503 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config" (OuterVolumeSpecName: "console-config") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:42:51.431291 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.431263 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75" (OuterVolumeSpecName: "kube-api-access-lwb75") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "kube-api-access-lwb75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:42:51.431416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.431400 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:42:51.431476 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.431420 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "250f34cb-7914-4e8f-a5f3-764bcc461ff7" (UID: "250f34cb-7914-4e8f-a5f3-764bcc461ff7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:42:51.530067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530040 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-service-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:51.530067 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530065 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:51.530448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530075 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:51.530448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530084 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwb75\" (UniqueName: \"kubernetes.io/projected/250f34cb-7914-4e8f-a5f3-764bcc461ff7-kube-api-access-lwb75\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:51.530448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530093 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250f34cb-7914-4e8f-a5f3-764bcc461ff7-console-oauth-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:51.530448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:51.530101 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250f34cb-7914-4e8f-a5f3-764bcc461ff7-oauth-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:42:52.116001 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.115959 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f6fc7b56b-hlc8h" Jul 16 13:42:52.116001 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.115980 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f6fc7b56b-hlc8h" event={"ID":"250f34cb-7914-4e8f-a5f3-764bcc461ff7","Type":"ContainerDied","Data":"54182bed1c250619394457d20a50b8f2dd34a35987c397175ef7f184c5ab26bf"} Jul 16 13:42:52.116261 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.116035 2563 scope.go:117] "RemoveContainer" containerID="d1185604d897610aa42a604be6c95100af13a624e122bf234297b3b802dc20d2" Jul 16 13:42:52.146229 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.146205 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:52.154113 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.154087 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f6fc7b56b-hlc8h"] Jul 16 13:42:52.473720 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:52.473628 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" path="/var/lib/kubelet/pods/250f34cb-7914-4e8f-a5f3-764bcc461ff7/volumes" Jul 16 13:42:53.060649 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:53.060611 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:42:59.950555 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:42:59.950510 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d5c5dd89d-w6t4r" podUID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" containerName="console" containerID="cri-o://74b139f8636c8fd58bd5764998794f3a7eab5c6375ce6c7cdb830c64cacbe008" gracePeriod=15 Jul 16 13:43:00.164958 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.164930 2563 generic.go:356] "Generic (PLEG): container finished" podID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" containerID="74b139f8636c8fd58bd5764998794f3a7eab5c6375ce6c7cdb830c64cacbe008" exitCode=0 Jul 16 13:43:00.165107 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.164987 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c5dd89d-w6t4r" event={"ID":"12b206cc-51b6-4d9e-bf69-a9a0af37001b","Type":"ContainerDied","Data":"74b139f8636c8fd58bd5764998794f3a7eab5c6375ce6c7cdb830c64cacbe008"} Jul 16 13:43:00.224842 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.224770 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:43:00.413468 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413432 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413468 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413477 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413539 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413567 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413581 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413690 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413602 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzgjh\" (UniqueName: \"kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.413910 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.413744 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle\") pod \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\" (UID: \"12b206cc-51b6-4d9e-bf69-a9a0af37001b\") " Jul 16 13:43:00.414041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.414015 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config" (OuterVolumeSpecName: "console-config") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:00.414298 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.414069 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:00.414406 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.414094 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca" (OuterVolumeSpecName: "service-ca") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:00.414406 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.414331 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:00.415762 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.415736 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh" (OuterVolumeSpecName: "kube-api-access-hzgjh") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "kube-api-access-hzgjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:43:00.415857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.415780 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:43:00.415857 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.415789 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "12b206cc-51b6-4d9e-bf69-a9a0af37001b" (UID: "12b206cc-51b6-4d9e-bf69-a9a0af37001b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:43:00.514559 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514523 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-oauth-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514559 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514555 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-service-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514559 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514566 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgjh\" (UniqueName: \"kubernetes.io/projected/12b206cc-51b6-4d9e-bf69-a9a0af37001b-kube-api-access-hzgjh\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514575 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-trusted-ca-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514583 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-oauth-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514591 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:00.514789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:00.514599 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12b206cc-51b6-4d9e-bf69-a9a0af37001b-console-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:01.170190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:01.170165 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c5dd89d-w6t4r" Jul 16 13:43:01.170608 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:01.170197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c5dd89d-w6t4r" event={"ID":"12b206cc-51b6-4d9e-bf69-a9a0af37001b","Type":"ContainerDied","Data":"014f4d6f98e0b7f284b04a885dd1de11bc6437494f2b946642cb6aaf2a72d1b1"} Jul 16 13:43:01.170608 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:01.170255 2563 scope.go:117] "RemoveContainer" containerID="74b139f8636c8fd58bd5764998794f3a7eab5c6375ce6c7cdb830c64cacbe008" Jul 16 13:43:01.187195 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:01.187168 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:43:01.189914 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:01.189890 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5c5dd89d-w6t4r"] Jul 16 13:43:02.472831 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:02.472794 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" path="/var/lib/kubelet/pods/12b206cc-51b6-4d9e-bf69-a9a0af37001b/volumes" Jul 16 13:43:09.412501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:09.412462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:09.513228 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:09.513190 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:11.161759 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.161682 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5888675787-zwg2n" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" containerName="console" containerID="cri-o://3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d" gracePeriod=15 Jul 16 13:43:11.423668 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.423611 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:43:11.485976 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.485940 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdhqv\" (UniqueName: \"kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.485976 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.485989 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486019 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486040 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486132 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486195 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config\") pod \"135e5c10-bd38-4464-8736-eec057cd9d20\" (UID: \"135e5c10-bd38-4464-8736-eec057cd9d20\") " Jul 16 13:43:11.486504 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486475 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:11.486569 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486518 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca" (OuterVolumeSpecName: "service-ca") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:11.486569 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486522 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config" (OuterVolumeSpecName: "console-config") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:11.486641 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.486577 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:43:11.488294 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.488269 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv" (OuterVolumeSpecName: "kube-api-access-gdhqv") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "kube-api-access-gdhqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:43:11.488294 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.488287 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:43:11.488494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.488338 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "135e5c10-bd38-4464-8736-eec057cd9d20" (UID: "135e5c10-bd38-4464-8736-eec057cd9d20"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:43:11.587201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587165 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-service-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587195 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-oauth-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587204 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-console-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587214 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587461 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587223 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/135e5c10-bd38-4464-8736-eec057cd9d20-trusted-ca-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587461 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587231 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/135e5c10-bd38-4464-8736-eec057cd9d20-console-oauth-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:11.587461 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:11.587240 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdhqv\" (UniqueName: \"kubernetes.io/projected/135e5c10-bd38-4464-8736-eec057cd9d20-kube-api-access-gdhqv\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:43:12.217414 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.217323 2563 generic.go:356] "Generic (PLEG): container finished" podID="135e5c10-bd38-4464-8736-eec057cd9d20" containerID="3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d" exitCode=0 Jul 16 13:43:12.217414 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.217370 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888675787-zwg2n" Jul 16 13:43:12.217959 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.217415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888675787-zwg2n" event={"ID":"135e5c10-bd38-4464-8736-eec057cd9d20","Type":"ContainerDied","Data":"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d"} Jul 16 13:43:12.217959 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.217468 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888675787-zwg2n" event={"ID":"135e5c10-bd38-4464-8736-eec057cd9d20","Type":"ContainerDied","Data":"fb2abb3163f9b2c9b74f118ce69a78cf82465738b94b355ebf8b8778e7ef5040"} Jul 16 13:43:12.217959 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.217490 2563 scope.go:117] "RemoveContainer" containerID="3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d" Jul 16 13:43:12.225765 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.225744 2563 scope.go:117] "RemoveContainer" containerID="3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d" Jul 16 13:43:12.226097 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:43:12.226075 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d\": container with ID starting with 3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d not found: ID does not exist" containerID="3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d" Jul 16 13:43:12.226149 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.226107 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d"} err="failed to get container status \"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d\": rpc error: code = NotFound desc = could not find container \"3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d\": container with ID starting with 3dc5ed364cc69d24d1cc402d070f59be88030f77f124cdfa9c05e8ac1000e67d not found: ID does not exist" Jul 16 13:43:12.238788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.238762 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:43:12.240562 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.240541 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5888675787-zwg2n"] Jul 16 13:43:12.473088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:12.472997 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" path="/var/lib/kubelet/pods/135e5c10-bd38-4464-8736-eec057cd9d20/volumes" Jul 16 13:43:41.387245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:41.387214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:41.488325 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:41.488290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:44.875819 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.875781 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:43:44.876319 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876290 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" containerName="console" Jul 16 13:43:44.876319 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876307 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" containerName="console" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876321 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" containerName="console" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876330 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" containerName="console" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876347 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerName="console" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876355 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerName="console" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876376 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerName="registry" Jul 16 13:43:44.876435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876384 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerName="registry" Jul 16 13:43:44.876698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876471 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b206cc-51b6-4d9e-bf69-a9a0af37001b" containerName="console" Jul 16 13:43:44.876698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876483 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="250f34cb-7914-4e8f-a5f3-764bcc461ff7" containerName="console" Jul 16 13:43:44.876698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876493 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="998e8a1b-eb66-46e9-a3f6-bf996d21c932" containerName="registry" Jul 16 13:43:44.876698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.876503 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="135e5c10-bd38-4464-8736-eec057cd9d20" containerName="console" Jul 16 13:43:44.878411 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.878392 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.892943 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.892912 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:43:44.945415 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945379 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945415 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945421 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945470 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945488 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.945633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.945593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:44.954549 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:44.954520 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:45.046551 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046551 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046776 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046776 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046776 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046966 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.046966 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.046840 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.047456 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.047432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.047566 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.047432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.047566 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.047547 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.047748 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.047728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.049163 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.049137 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.049273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.049242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.061360 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.061328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:45.061719 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.061693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87\") pod \"console-7c757b96fc-l2rqx\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.118004 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.117961 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:45.187713 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.187623 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:45.323530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.323497 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:43:45.327238 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:43:45.327206 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc195628_c9ef_47a2_a9d0_7213d52d57f6.slice/crio-ce80551c773fb9f4584ea72e2e0ef6676ba27b7fa5faa50d41f60356a12b938b WatchSource:0}: Error finding container ce80551c773fb9f4584ea72e2e0ef6676ba27b7fa5faa50d41f60356a12b938b: Status 404 returned error can't find the container with id ce80551c773fb9f4584ea72e2e0ef6676ba27b7fa5faa50d41f60356a12b938b Jul 16 13:43:45.341096 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.341073 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c757b96fc-l2rqx" event={"ID":"fc195628-c9ef-47a2-a9d0-7213d52d57f6","Type":"ContainerStarted","Data":"ce80551c773fb9f4584ea72e2e0ef6676ba27b7fa5faa50d41f60356a12b938b"} Jul 16 13:43:45.681911 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.681850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:45.759665 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:45.759630 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:46.346277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:46.346245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c757b96fc-l2rqx" event={"ID":"fc195628-c9ef-47a2-a9d0-7213d52d57f6","Type":"ContainerStarted","Data":"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3"} Jul 16 13:43:46.372387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:46.372331 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c757b96fc-l2rqx" podStartSLOduration=2.372314514 podStartE2EDuration="2.372314514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:43:46.371940902 +0000 UTC m=+206.320057628" watchObservedRunningTime="2026-07-16 13:43:46.372314514 +0000 UTC m=+206.320431220" Jul 16 13:43:53.204286 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:53.204255 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:53.319276 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:53.319234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:55.188306 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.188261 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:55.188694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.188343 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:55.193206 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.193182 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:55.397482 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.397450 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:43:55.464464 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.464376 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:43:55.617443 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.617408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:55.759494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.759405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:43:55.846532 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:43:55.846501 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:44:09.470274 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:09.470242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:44:09.577654 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:09.577604 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:44:12.888343 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:12.888312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:44:12.996336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:12.996297 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:44:20.491494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.491429 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dbcf88664-q5qhm" podUID="e04beb06-dd59-45c4-b40a-4fee126a6566" containerName="console" containerID="cri-o://287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2" gracePeriod=15 Jul 16 13:44:20.741717 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.741642 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:44:20.834640 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834602 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834640 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834639 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834657 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834744 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834781 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl4s2\" (UniqueName: \"kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.834867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.834807 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config\") pod \"e04beb06-dd59-45c4-b40a-4fee126a6566\" (UID: \"e04beb06-dd59-45c4-b40a-4fee126a6566\") " Jul 16 13:44:20.835196 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.835165 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca" (OuterVolumeSpecName: "service-ca") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:44:20.835196 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.835173 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:44:20.835327 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.835178 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:44:20.835404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.835388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config" (OuterVolumeSpecName: "console-config") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:44:20.836846 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.836811 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:44:20.837060 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.836866 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:44:20.837060 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.836969 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2" (OuterVolumeSpecName: "kube-api-access-sl4s2") pod "e04beb06-dd59-45c4-b40a-4fee126a6566" (UID: "e04beb06-dd59-45c4-b40a-4fee126a6566"). InnerVolumeSpecName "kube-api-access-sl4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:44:20.936382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936331 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-service-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936376 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936389 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-oauth-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936398 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-trusted-ca-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936654 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936406 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e04beb06-dd59-45c4-b40a-4fee126a6566-console-oauth-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936654 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936415 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4s2\" (UniqueName: \"kubernetes.io/projected/e04beb06-dd59-45c4-b40a-4fee126a6566-kube-api-access-sl4s2\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:20.936654 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:20.936423 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e04beb06-dd59-45c4-b40a-4fee126a6566-console-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:44:21.482798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.482762 2563 generic.go:356] "Generic (PLEG): container finished" podID="e04beb06-dd59-45c4-b40a-4fee126a6566" containerID="287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2" exitCode=0 Jul 16 13:44:21.482997 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.482806 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf88664-q5qhm" Jul 16 13:44:21.482997 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.482816 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf88664-q5qhm" event={"ID":"e04beb06-dd59-45c4-b40a-4fee126a6566","Type":"ContainerDied","Data":"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2"} Jul 16 13:44:21.482997 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.482856 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf88664-q5qhm" event={"ID":"e04beb06-dd59-45c4-b40a-4fee126a6566","Type":"ContainerDied","Data":"7beaf8bb075510389212fd6755097b787cdc5dbce75d0548753a3a9d5fb86b97"} Jul 16 13:44:21.482997 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.482897 2563 scope.go:117] "RemoveContainer" containerID="287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2" Jul 16 13:44:21.491533 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.491511 2563 scope.go:117] "RemoveContainer" containerID="287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2" Jul 16 13:44:21.491824 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:44:21.491807 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2\": container with ID starting with 287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2 not found: ID does not exist" containerID="287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2" Jul 16 13:44:21.491864 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.491834 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2"} err="failed to get container status \"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2\": rpc error: code = NotFound desc = could not find container \"287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2\": container with ID starting with 287f0817d6db32728bd48985ca4d3060734d6f5a9ea336213eaec740631ee3f2 not found: ID does not exist" Jul 16 13:44:21.508084 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.508051 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:44:21.512985 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:21.512958 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dbcf88664-q5qhm"] Jul 16 13:44:22.472773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:44:22.472737 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e04beb06-dd59-45c4-b40a-4fee126a6566" path="/var/lib/kubelet/pods/e04beb06-dd59-45c4-b40a-4fee126a6566/volumes" Jul 16 13:45:09.764013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:09.763977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:13.596613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:13.596566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:13.700208 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:13.700170 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:14.297686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.297650 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf"] Jul 16 13:45:14.297998 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.297985 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04beb06-dd59-45c4-b40a-4fee126a6566" containerName="console" Jul 16 13:45:14.298051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.297999 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04beb06-dd59-45c4-b40a-4fee126a6566" containerName="console" Jul 16 13:45:14.298051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.298039 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04beb06-dd59-45c4-b40a-4fee126a6566" containerName="console" Jul 16 13:45:14.299993 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.299978 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.302223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.302202 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 13:45:14.302352 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.302221 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 13:45:14.302352 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.302252 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcm6s\"" Jul 16 13:45:14.311766 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.311743 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf"] Jul 16 13:45:14.450996 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.450958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wgv\" (UniqueName: \"kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.450996 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.451002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.451216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.451025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.552210 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.552134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.552210 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.552174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.552387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.552234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wgv\" (UniqueName: \"kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.552594 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.552570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.552660 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.552602 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.560292 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.560266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wgv\" (UniqueName: \"kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.609756 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.609699 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:14.736433 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:14.736402 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf"] Jul 16 13:45:14.739233 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:45:14.739208 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e35aa8_5b8a_4034_a8b6_ed87fda52ae2.slice/crio-f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9 WatchSource:0}: Error finding container f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9: Status 404 returned error can't find the container with id f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9 Jul 16 13:45:15.685482 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:15.685432 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" event={"ID":"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2","Type":"ContainerStarted","Data":"f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9"} Jul 16 13:45:19.055960 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:19.055908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:19.120278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:19.120242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:20.595385 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:20.595356 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:45:20.595845 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:20.595604 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:45:20.601228 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:20.601211 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 13:45:21.712678 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:21.712641 2563 generic.go:356] "Generic (PLEG): container finished" podID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerID="5d8a1dabfc77a7069354402ac5ccc433b1799dae3fb6ce32447ba3fcf9617e5e" exitCode=0 Jul 16 13:45:21.713109 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:21.712732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" event={"ID":"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2","Type":"ContainerDied","Data":"5d8a1dabfc77a7069354402ac5ccc433b1799dae3fb6ce32447ba3fcf9617e5e"} Jul 16 13:45:21.713860 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:21.713843 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:45:23.722516 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:23.722481 2563 generic.go:356] "Generic (PLEG): container finished" podID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerID="0f5773a6678a4187fa647f557a761dacf99f3f345610b140fce674deff281841" exitCode=0 Jul 16 13:45:23.722516 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:23.722521 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" event={"ID":"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2","Type":"ContainerDied","Data":"0f5773a6678a4187fa647f557a761dacf99f3f345610b140fce674deff281841"} Jul 16 13:45:32.761466 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:32.761434 2563 generic.go:356] "Generic (PLEG): container finished" podID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerID="319ce269d820583b5245db091c05f2acf00d5995253fe29447a64b1c51617e44" exitCode=0 Jul 16 13:45:32.761846 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:32.761486 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" event={"ID":"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2","Type":"ContainerDied","Data":"319ce269d820583b5245db091c05f2acf00d5995253fe29447a64b1c51617e44"} Jul 16 13:45:33.891262 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.891238 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:33.912176 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.912149 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6wgv\" (UniqueName: \"kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv\") pod \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " Jul 16 13:45:33.912176 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.912182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util\") pod \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " Jul 16 13:45:33.912420 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.912249 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle\") pod \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\" (UID: \"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2\") " Jul 16 13:45:33.913099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.913073 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle" (OuterVolumeSpecName: "bundle") pod "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" (UID: "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:45:33.914756 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.914728 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv" (OuterVolumeSpecName: "kube-api-access-v6wgv") pod "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" (UID: "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2"). InnerVolumeSpecName "kube-api-access-v6wgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:45:33.916948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:33.916922 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util" (OuterVolumeSpecName: "util") pod "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" (UID: "47e35aa8-5b8a-4034-a8b6-ed87fda52ae2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:45:34.013553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.013511 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wgv\" (UniqueName: \"kubernetes.io/projected/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-kube-api-access-v6wgv\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:34.013553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.013545 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:34.013765 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.013577 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e35aa8-5b8a-4034-a8b6-ed87fda52ae2-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:34.771934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.771887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" event={"ID":"47e35aa8-5b8a-4034-a8b6-ed87fda52ae2","Type":"ContainerDied","Data":"f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9"} Jul 16 13:45:34.772108 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.771955 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f02c09bf46e366ed7eea5fdc65316e24a108f31cfd6f8f110cbe18f7111f18e9" Jul 16 13:45:34.772108 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:34.771914 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406m4bf" Jul 16 13:45:40.558400 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:40.558365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:41.371928 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:41.371896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:42.466302 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466264 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx"] Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466549 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="extract" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466559 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="extract" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466569 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="pull" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466575 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="pull" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466596 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="util" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466602 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="util" Jul 16 13:45:42.466694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.466653 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e35aa8-5b8a-4034-a8b6-ed87fda52ae2" containerName="extract" Jul 16 13:45:42.469351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.469332 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.472103 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.472074 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 16 13:45:42.472248 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.472207 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 16 13:45:42.472336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.472317 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z7klj\"" Jul 16 13:45:42.491088 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.491056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx"] Jul 16 13:45:42.577208 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.577153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415747e6-8953-42e0-bea6-f83daf28c261-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.577208 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.577217 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h8l\" (UniqueName: \"kubernetes.io/projected/415747e6-8953-42e0-bea6-f83daf28c261-kube-api-access-62h8l\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.677850 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.677805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415747e6-8953-42e0-bea6-f83daf28c261-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.677850 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.677861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h8l\" (UniqueName: \"kubernetes.io/projected/415747e6-8953-42e0-bea6-f83daf28c261-kube-api-access-62h8l\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.678211 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.678191 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415747e6-8953-42e0-bea6-f83daf28c261-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.691114 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.691092 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h8l\" (UniqueName: \"kubernetes.io/projected/415747e6-8953-42e0-bea6-f83daf28c261-kube-api-access-62h8l\") pod \"cert-manager-operator-controller-manager-55c444bf86-g84gx\" (UID: \"415747e6-8953-42e0-bea6-f83daf28c261\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.778833 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.778793 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" Jul 16 13:45:42.919406 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:42.919350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx"] Jul 16 13:45:42.921732 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:45:42.921702 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415747e6_8953_42e0_bea6_f83daf28c261.slice/crio-67e9996f4e55dd794dc3434b9eb6eceb7df98cdad3eebb2d25b6210fb2bd465a WatchSource:0}: Error finding container 67e9996f4e55dd794dc3434b9eb6eceb7df98cdad3eebb2d25b6210fb2bd465a: Status 404 returned error can't find the container with id 67e9996f4e55dd794dc3434b9eb6eceb7df98cdad3eebb2d25b6210fb2bd465a Jul 16 13:45:43.805546 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:43.805504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" event={"ID":"415747e6-8953-42e0-bea6-f83daf28c261","Type":"ContainerStarted","Data":"67e9996f4e55dd794dc3434b9eb6eceb7df98cdad3eebb2d25b6210fb2bd465a"} Jul 16 13:45:43.999308 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:43.999258 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:44.100539 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:44.100442 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:45.593664 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:45.593630 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:45.688296 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:45.688261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:45.817387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:45.817352 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" event={"ID":"415747e6-8953-42e0-bea6-f83daf28c261","Type":"ContainerStarted","Data":"ed00dbb14c182536e4995e03cc588548df114b107e485938c535d19b7ce3d2d7"} Jul 16 13:45:45.844436 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:45.844328 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g84gx" podStartSLOduration=2.022483083 podStartE2EDuration="3.844311128s" totalImagesPullingTime="1.821828045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:45:45.843954036 +0000 UTC m=+325.792070761" watchObservedRunningTime="2026-07-16 13:45:45.844311128 +0000 UTC m=+325.792427834" Jul 16 13:45:46.324223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.324188 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:46.654063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.653976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:46.874598 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.874559 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt"] Jul 16 13:45:46.877634 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.877618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:46.879897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.879840 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 13:45:46.879897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.879893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 13:45:46.880101 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.879937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcm6s\"" Jul 16 13:45:46.887096 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.887075 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt"] Jul 16 13:45:46.915106 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.915020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tfn\" (UniqueName: \"kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:46.915273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.915131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:46.915273 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:46.915194 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.016200 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.016159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.016388 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.016215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.016388 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.016271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tfn\" (UniqueName: \"kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.016577 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.016558 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.016630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.016584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.024899 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.024856 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tfn\" (UniqueName: \"kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.187497 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.187413 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:47.331074 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.331036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt"] Jul 16 13:45:47.334116 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:45:47.334088 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a3d21e_2874_445d_9cc3_9bcfd6809257.slice/crio-62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1 WatchSource:0}: Error finding container 62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1: Status 404 returned error can't find the container with id 62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1 Jul 16 13:45:47.827482 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.827453 2563 generic.go:356] "Generic (PLEG): container finished" podID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerID="de25e8894efe0670ed8ab6990e8598bfd4c0bd4f384d0c82246d3ff7ecfecbd0" exitCode=0 Jul 16 13:45:47.827897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.827525 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" event={"ID":"83a3d21e-2874-445d-9cc3-9bcfd6809257","Type":"ContainerDied","Data":"de25e8894efe0670ed8ab6990e8598bfd4c0bd4f384d0c82246d3ff7ecfecbd0"} Jul 16 13:45:47.827897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:47.827558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" event={"ID":"83a3d21e-2874-445d-9cc3-9bcfd6809257","Type":"ContainerStarted","Data":"62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1"} Jul 16 13:45:48.000238 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:48.000201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:48.398798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:48.398757 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:49.237918 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.237870 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4687z"] Jul 16 13:45:49.246353 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.246327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.248883 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.248847 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 13:45:49.249485 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.249464 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 13:45:49.250188 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.250172 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rfv6k\"" Jul 16 13:45:49.252351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.252332 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4687z"] Jul 16 13:45:49.337007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.336966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrxb\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-kube-api-access-dvrxb\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.337171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.337113 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.438491 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.438450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrxb\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-kube-api-access-dvrxb\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.438693 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.438547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.447207 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.447175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.447505 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.447481 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrxb\" (UniqueName: \"kubernetes.io/projected/1f0ffdec-d779-43c6-a6cb-06d78d141ee7-kube-api-access-dvrxb\") pod \"cert-manager-cainjector-559cdd8c87-4687z\" (UID: \"1f0ffdec-d779-43c6-a6cb-06d78d141ee7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.568183 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.568142 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" Jul 16 13:45:49.831949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:49.831923 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4687z"] Jul 16 13:45:49.835818 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:45:49.835789 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f0ffdec_d779_43c6_a6cb_06d78d141ee7.slice/crio-b0a3592893873a0bf29c0f346f2af87cf8b99b053001e9c4040ae539016e37e0 WatchSource:0}: Error finding container b0a3592893873a0bf29c0f346f2af87cf8b99b053001e9c4040ae539016e37e0: Status 404 returned error can't find the container with id b0a3592893873a0bf29c0f346f2af87cf8b99b053001e9c4040ae539016e37e0 Jul 16 13:45:50.846059 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:50.846027 2563 generic.go:356] "Generic (PLEG): container finished" podID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerID="c78420a3017edacaaf0750a7e4ab7898a7f135a8a9669dee89cfab3da39f3729" exitCode=0 Jul 16 13:45:50.846487 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:50.846122 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" event={"ID":"83a3d21e-2874-445d-9cc3-9bcfd6809257","Type":"ContainerDied","Data":"c78420a3017edacaaf0750a7e4ab7898a7f135a8a9669dee89cfab3da39f3729"} Jul 16 13:45:50.847442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:50.847420 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" event={"ID":"1f0ffdec-d779-43c6-a6cb-06d78d141ee7","Type":"ContainerStarted","Data":"b0a3592893873a0bf29c0f346f2af87cf8b99b053001e9c4040ae539016e37e0"} Jul 16 13:45:51.007345 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:51.007313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:51.071534 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:51.071492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:51.854786 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:51.854745 2563 generic.go:356] "Generic (PLEG): container finished" podID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerID="4d99dd7d83cb5c526bfb7a895ac5b0ea11390072b63598d81b9926b4fb5a9cab" exitCode=0 Jul 16 13:45:51.855268 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:51.854866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" event={"ID":"83a3d21e-2874-445d-9cc3-9bcfd6809257","Type":"ContainerDied","Data":"4d99dd7d83cb5c526bfb7a895ac5b0ea11390072b63598d81b9926b4fb5a9cab"} Jul 16 13:45:52.860718 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:52.860675 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" event={"ID":"1f0ffdec-d779-43c6-a6cb-06d78d141ee7","Type":"ContainerStarted","Data":"30d6fe5e80c0751ae56ac9f34218388935909403b7aacc4d927401347959b286"} Jul 16 13:45:52.875866 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:52.875809 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4687z" podStartSLOduration=1.348437106 podStartE2EDuration="3.87578938s" totalImagesPullingTime="2.527352274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:45:52.875508619 +0000 UTC m=+332.823625327" watchObservedRunningTime="2026-07-16 13:45:52.87578938 +0000 UTC m=+332.823906088" Jul 16 13:45:52.996750 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:52.996724 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:53.070398 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.070361 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util\") pod \"83a3d21e-2874-445d-9cc3-9bcfd6809257\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " Jul 16 13:45:53.070599 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.070468 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle\") pod \"83a3d21e-2874-445d-9cc3-9bcfd6809257\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " Jul 16 13:45:53.070599 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.070500 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89tfn\" (UniqueName: \"kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn\") pod \"83a3d21e-2874-445d-9cc3-9bcfd6809257\" (UID: \"83a3d21e-2874-445d-9cc3-9bcfd6809257\") " Jul 16 13:45:53.070918 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.070869 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle" (OuterVolumeSpecName: "bundle") pod "83a3d21e-2874-445d-9cc3-9bcfd6809257" (UID: "83a3d21e-2874-445d-9cc3-9bcfd6809257"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:45:53.072564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.072542 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn" (OuterVolumeSpecName: "kube-api-access-89tfn") pod "83a3d21e-2874-445d-9cc3-9bcfd6809257" (UID: "83a3d21e-2874-445d-9cc3-9bcfd6809257"). InnerVolumeSpecName "kube-api-access-89tfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:45:53.075120 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.075094 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util" (OuterVolumeSpecName: "util") pod "83a3d21e-2874-445d-9cc3-9bcfd6809257" (UID: "83a3d21e-2874-445d-9cc3-9bcfd6809257"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:45:53.171694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.171603 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:53.171694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.171632 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83a3d21e-2874-445d-9cc3-9bcfd6809257-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:53.171694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.171640 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89tfn\" (UniqueName: \"kubernetes.io/projected/83a3d21e-2874-445d-9cc3-9bcfd6809257-kube-api-access-89tfn\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:45:53.866411 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.866374 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" event={"ID":"83a3d21e-2874-445d-9cc3-9bcfd6809257","Type":"ContainerDied","Data":"62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1"} Jul 16 13:45:53.866411 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.866407 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdkwpt" Jul 16 13:45:53.866819 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:53.866427 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b09da2efc6236fed61162b7b7ca002458c74ac4e31fd88ed9455fbddf63bb1" Jul 16 13:45:55.190634 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.190601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:55.962680 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.962642 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9hb46"] Jul 16 13:45:55.962995 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.962982 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="pull" Jul 16 13:45:55.963048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.962998 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="pull" Jul 16 13:45:55.963048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.963022 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="extract" Jul 16 13:45:55.963048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.963029 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="extract" Jul 16 13:45:55.963048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.963039 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="util" Jul 16 13:45:55.963048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.963046 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="util" Jul 16 13:45:55.963186 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.963106 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a3d21e-2874-445d-9cc3-9bcfd6809257" containerName="extract" Jul 16 13:45:55.966843 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.966827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:55.968942 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.968923 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jr84g\"" Jul 16 13:45:55.975576 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:55.975550 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9hb46"] Jul 16 13:45:56.095923 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.095859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrnf\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-kube-api-access-mbrnf\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.096098 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.095932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-bound-sa-token\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.196953 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.196918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrnf\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-kube-api-access-mbrnf\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.196953 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.196954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-bound-sa-token\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.205574 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.205550 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-bound-sa-token\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.205821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.205799 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrnf\" (UniqueName: \"kubernetes.io/projected/c91892f1-1e28-48f2-a1d7-ec84303bf79b-kube-api-access-mbrnf\") pod \"cert-manager-55b4d954d5-9hb46\" (UID: \"c91892f1-1e28-48f2-a1d7-ec84303bf79b\") " pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.276366 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.276341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9hb46" Jul 16 13:45:56.406211 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.406188 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9hb46"] Jul 16 13:45:56.408919 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:45:56.408889 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc91892f1_1e28_48f2_a1d7_ec84303bf79b.slice/crio-512d42519b3af7f559782cbdbe20739bd195014803d0ab7ef15bc90c1fc33ac9 WatchSource:0}: Error finding container 512d42519b3af7f559782cbdbe20739bd195014803d0ab7ef15bc90c1fc33ac9: Status 404 returned error can't find the container with id 512d42519b3af7f559782cbdbe20739bd195014803d0ab7ef15bc90c1fc33ac9 Jul 16 13:45:56.880924 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.880859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9hb46" event={"ID":"c91892f1-1e28-48f2-a1d7-ec84303bf79b","Type":"ContainerStarted","Data":"08465801c6fc44586083e93d6c0782875772a898bea7ea06b1df0ae1b76ec8b2"} Jul 16 13:45:56.881104 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.880937 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9hb46" event={"ID":"c91892f1-1e28-48f2-a1d7-ec84303bf79b","Type":"ContainerStarted","Data":"512d42519b3af7f559782cbdbe20739bd195014803d0ab7ef15bc90c1fc33ac9"} Jul 16 13:45:56.898484 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:56.898423 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9hb46" podStartSLOduration=1.898407449 podStartE2EDuration="1.898407449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:45:56.89619012 +0000 UTC m=+336.844306826" watchObservedRunningTime="2026-07-16 13:45:56.898407449 +0000 UTC m=+336.846524156" Jul 16 13:45:58.683633 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:58.683599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:45:58.783621 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:45:58.783587 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:03.093292 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:03.093255 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:04.484431 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:04.484397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:05.914096 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.914060 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx"] Jul 16 13:46:05.916708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.916683 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:05.919042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.919023 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 13:46:05.919145 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.919023 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 13:46:05.919687 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.919671 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dcm6s\"" Jul 16 13:46:05.924775 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:05.924748 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx"] Jul 16 13:46:06.082746 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.082702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.082949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.082792 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.082949 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.082864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mfp\" (UniqueName: \"kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.146717 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.146655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:06.184082 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.183994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.184082 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.184083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mfp\" (UniqueName: \"kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.184265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.184113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.184400 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.184382 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.184464 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.184442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.196386 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.196361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mfp\" (UniqueName: \"kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.227378 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.227339 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:06.361686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.361663 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx"] Jul 16 13:46:06.363742 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:06.363715 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae628f2_3cd6_44a9_bdf2_b9990433d1f0.slice/crio-7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c WatchSource:0}: Error finding container 7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c: Status 404 returned error can't find the container with id 7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c Jul 16 13:46:06.925165 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.925131 2563 generic.go:356] "Generic (PLEG): container finished" podID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerID="4ffdf797dbd23025e6e00024b92a1979996785200c449fa5cab3d79c0f600517" exitCode=0 Jul 16 13:46:06.925560 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.925203 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" event={"ID":"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0","Type":"ContainerDied","Data":"4ffdf797dbd23025e6e00024b92a1979996785200c449fa5cab3d79c0f600517"} Jul 16 13:46:06.925560 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:06.925239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" event={"ID":"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0","Type":"ContainerStarted","Data":"7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c"} Jul 16 13:46:07.931523 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:07.931490 2563 generic.go:356] "Generic (PLEG): container finished" podID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerID="7ded9b486cf50f02ed0e0f40bfd639d0cca773e1140839c055a17684d762c349" exitCode=0 Jul 16 13:46:07.931928 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:07.931555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" event={"ID":"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0","Type":"ContainerDied","Data":"7ded9b486cf50f02ed0e0f40bfd639d0cca773e1140839c055a17684d762c349"} Jul 16 13:46:08.938317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:08.938281 2563 generic.go:356] "Generic (PLEG): container finished" podID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerID="0171936b2928bcefc605882520fee2fbb603e7163e518b86a384befc09798162" exitCode=0 Jul 16 13:46:08.938694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:08.938359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" event={"ID":"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0","Type":"ContainerDied","Data":"0171936b2928bcefc605882520fee2fbb603e7163e518b86a384befc09798162"} Jul 16 13:46:10.071235 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.071208 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:10.219014 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.218917 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle\") pod \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " Jul 16 13:46:10.219014 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.218955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util\") pod \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " Jul 16 13:46:10.219014 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.218993 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2mfp\" (UniqueName: \"kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp\") pod \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\" (UID: \"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0\") " Jul 16 13:46:10.219715 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.219687 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle" (OuterVolumeSpecName: "bundle") pod "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" (UID: "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:46:10.221191 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.221171 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp" (OuterVolumeSpecName: "kube-api-access-k2mfp") pod "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" (UID: "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0"). InnerVolumeSpecName "kube-api-access-k2mfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:46:10.224686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.224666 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util" (OuterVolumeSpecName: "util") pod "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" (UID: "6ae628f2-3cd6-44a9-bdf2-b9990433d1f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:46:10.319583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.319548 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:10.319583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.319573 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:10.319583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.319582 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2mfp\" (UniqueName: \"kubernetes.io/projected/6ae628f2-3cd6-44a9-bdf2-b9990433d1f0-kube-api-access-k2mfp\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:10.949321 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.949294 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" Jul 16 13:46:10.949489 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.949321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhh6sx" event={"ID":"6ae628f2-3cd6-44a9-bdf2-b9990433d1f0","Type":"ContainerDied","Data":"7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c"} Jul 16 13:46:10.949489 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:10.949363 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7938ec0f32640cffcbed433f9772273e5418f5fa1fe20bd95a32a2fb408a617c" Jul 16 13:46:15.701599 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:15.701523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:15.801545 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:15.801509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:21.682187 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:21.682146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:21.807336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:21.807302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:22.274192 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.274154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:22.834455 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.834422 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:22.964760 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.964723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:22.987515 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987480 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq"] Jul 16 13:46:22.987801 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987789 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="util" Jul 16 13:46:22.987844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987802 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="util" Jul 16 13:46:22.987844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987816 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="extract" Jul 16 13:46:22.987844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987822 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="extract" Jul 16 13:46:22.987844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987836 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="pull" Jul 16 13:46:22.987844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987841 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="pull" Jul 16 13:46:22.988011 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.987925 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae628f2-3cd6-44a9-bdf2-b9990433d1f0" containerName="extract" Jul 16 13:46:22.990017 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.989992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:22.994429 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.994407 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 13:46:22.994535 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.994448 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 13:46:22.994810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.994796 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvv78\"" Jul 16 13:46:22.996737 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.996403 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 13:46:22.996737 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:22.996422 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 13:46:23.040784 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.040755 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq"] Jul 16 13:46:23.119704 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.119615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-manifests\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.119704 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.119645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-tmp\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.119704 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.119691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.119967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.119716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.119967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.119770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjdv\" (UniqueName: \"kubernetes.io/projected/c06c9634-bc6c-4670-8311-47245d2731ba-kube-api-access-zkjdv\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221139 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjdv\" (UniqueName: \"kubernetes.io/projected/c06c9634-bc6c-4670-8311-47245d2731ba-kube-api-access-zkjdv\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221322 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-manifests\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221322 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221213 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-tmp\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221322 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221416 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.221530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.221512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-manifests\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.223548 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.223526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c06c9634-bc6c-4670-8311-47245d2731ba-tmp\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.223676 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.223653 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.223794 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.223777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c06c9634-bc6c-4670-8311-47245d2731ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.233435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.233410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:23.239623 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.239606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjdv\" (UniqueName: \"kubernetes.io/projected/c06c9634-bc6c-4670-8311-47245d2731ba-kube-api-access-zkjdv\") pod \"opendatahub-operator-controller-manager-558d6dc665-7fzrq\" (UID: \"c06c9634-bc6c-4670-8311-47245d2731ba\") " pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.299842 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.299810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:23.325607 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.325574 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:23.425138 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.425094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:23.434149 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.434047 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq"] Jul 16 13:46:23.436477 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:23.436444 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06c9634_bc6c_4670_8311_47245d2731ba.slice/crio-25f3a836b07f1a7e7f1dc55c884b13fa2ae6ba817588dccfb21e97a05ef8aa81 WatchSource:0}: Error finding container 25f3a836b07f1a7e7f1dc55c884b13fa2ae6ba817588dccfb21e97a05ef8aa81: Status 404 returned error can't find the container with id 25f3a836b07f1a7e7f1dc55c884b13fa2ae6ba817588dccfb21e97a05ef8aa81 Jul 16 13:46:23.514754 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.514723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:23.621889 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.621850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:23.717125 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:23.717033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:24.009945 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:24.009832 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" event={"ID":"c06c9634-bc6c-4670-8311-47245d2731ba","Type":"ContainerStarted","Data":"25f3a836b07f1a7e7f1dc55c884b13fa2ae6ba817588dccfb21e97a05ef8aa81"} Jul 16 13:46:27.027469 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:27.027438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" event={"ID":"c06c9634-bc6c-4670-8311-47245d2731ba","Type":"ContainerStarted","Data":"37e2bb78c045b3b7393ec3cc4938477875455e9907f1de526f4a5f691a5e9b24"} Jul 16 13:46:28.033941 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:28.033909 2563 generic.go:356] "Generic (PLEG): container finished" podID="c06c9634-bc6c-4670-8311-47245d2731ba" containerID="37e2bb78c045b3b7393ec3cc4938477875455e9907f1de526f4a5f691a5e9b24" exitCode=0 Jul 16 13:46:28.034319 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:28.033972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" event={"ID":"c06c9634-bc6c-4670-8311-47245d2731ba","Type":"ContainerDied","Data":"37e2bb78c045b3b7393ec3cc4938477875455e9907f1de526f4a5f691a5e9b24"} Jul 16 13:46:29.040404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:29.040367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" event={"ID":"c06c9634-bc6c-4670-8311-47245d2731ba","Type":"ContainerStarted","Data":"3757fe89cc17ac018b3de69d1d3c14fb740860f7e09059c1f14e0f915bb94151"} Jul 16 13:46:29.040772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:29.040442 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:29.077322 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:29.077265 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" podStartSLOduration=2.580830323 podStartE2EDuration="7.077249248s" totalImagesPullingTime="3.488291448s" totalInitContainerRuntime="1.008127477s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:46:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:46:29.075727466 +0000 UTC m=+369.023844174" watchObservedRunningTime="2026-07-16 13:46:29.077249248 +0000 UTC m=+369.025365955" Jul 16 13:46:32.534134 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:32.534096 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:32.634539 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:32.634507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:40.046444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:40.046412 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-558d6dc665-7fzrq" Jul 16 13:46:43.173599 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.173563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:43.315276 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.315242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:43.479798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.479708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:43.625887 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.625850 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b75db759-xwgcj"] Jul 16 13:46:43.629069 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.629052 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.631793 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.631768 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xlc6s\"" Jul 16 13:46:43.631954 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.631774 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 13:46:43.632630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.632614 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 13:46:43.646124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.646096 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b75db759-xwgcj"] Jul 16 13:46:43.684739 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.684699 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7204850-f591-4438-8c51-ace01411191a-tmp\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.684933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.684781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw67t\" (UniqueName: \"kubernetes.io/projected/b7204850-f591-4438-8c51-ace01411191a-kube-api-access-xw67t\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.684933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.684865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7204850-f591-4438-8c51-ace01411191a-tls-certs\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.770708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.770676 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:43.785767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.785730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw67t\" (UniqueName: \"kubernetes.io/projected/b7204850-f591-4438-8c51-ace01411191a-kube-api-access-xw67t\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.785967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.785916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7204850-f591-4438-8c51-ace01411191a-tls-certs\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.785967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.785958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7204850-f591-4438-8c51-ace01411191a-tmp\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.788195 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.788172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7204850-f591-4438-8c51-ace01411191a-tmp\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.788393 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.788377 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7204850-f591-4438-8c51-ace01411191a-tls-certs\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.810073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.810038 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw67t\" (UniqueName: \"kubernetes.io/projected/b7204850-f591-4438-8c51-ace01411191a-kube-api-access-xw67t\") pod \"kube-auth-proxy-b75db759-xwgcj\" (UID: \"b7204850-f591-4438-8c51-ace01411191a\") " pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:43.939564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:43.939511 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" Jul 16 13:46:44.076459 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.076423 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fctwd"] Jul 16 13:46:44.079952 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.079936 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.082354 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.082333 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 13:46:44.082494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.082478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9vzv4\"" Jul 16 13:46:44.088027 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.088003 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b75db759-xwgcj"] Jul 16 13:46:44.090925 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:44.090901 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7204850_f591_4438_8c51_ace01411191a.slice/crio-8521e330209adc283ee7f5f90b3cd30972caebe699f4be75efb459044b4ae215 WatchSource:0}: Error finding container 8521e330209adc283ee7f5f90b3cd30972caebe699f4be75efb459044b4ae215: Status 404 returned error can't find the container with id 8521e330209adc283ee7f5f90b3cd30972caebe699f4be75efb459044b4ae215 Jul 16 13:46:44.094985 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.094965 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fctwd"] Jul 16 13:46:44.108655 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.108619 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" event={"ID":"b7204850-f591-4438-8c51-ace01411191a","Type":"ContainerStarted","Data":"8521e330209adc283ee7f5f90b3cd30972caebe699f4be75efb459044b4ae215"} Jul 16 13:46:44.182812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.182780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:44.188758 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.188736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.188846 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.188794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcd4\" (UniqueName: \"kubernetes.io/projected/bb228b98-3ec5-4802-a27e-8993fa0c68e9-kube-api-access-7dcd4\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.285051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.284959 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:44.289936 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.289914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.290053 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.289983 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcd4\" (UniqueName: \"kubernetes.io/projected/bb228b98-3ec5-4802-a27e-8993fa0c68e9-kube-api-access-7dcd4\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.290098 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:46:44.290045 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 16 13:46:44.290131 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:46:44.290123 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert podName:bb228b98-3ec5-4802-a27e-8993fa0c68e9 nodeName:}" failed. No retries permitted until 2026-07-16 13:46:44.790104313 +0000 UTC m=+384.738220999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert") pod "odh-model-controller-7998c67dc4-fctwd" (UID: "bb228b98-3ec5-4802-a27e-8993fa0c68e9") : secret "odh-model-controller-webhook-cert" not found Jul 16 13:46:44.300971 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.300945 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcd4\" (UniqueName: \"kubernetes.io/projected/bb228b98-3ec5-4802-a27e-8993fa0c68e9-kube-api-access-7dcd4\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.386733 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.386703 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:44.484424 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.484391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:44.589254 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.589163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:44.796892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.796843 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.799183 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.799163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb228b98-3ec5-4802-a27e-8993fa0c68e9-cert\") pod \"odh-model-controller-7998c67dc4-fctwd\" (UID: \"bb228b98-3ec5-4802-a27e-8993fa0c68e9\") " pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:44.991100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:44.991011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:45.158229 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.158198 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fctwd"] Jul 16 13:46:45.162278 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:45.162245 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb228b98_3ec5_4802_a27e_8993fa0c68e9.slice/crio-137654b2a278ba1487854df4e3dd2035a34e10a0aa051fac14d63e2451ad77f0 WatchSource:0}: Error finding container 137654b2a278ba1487854df4e3dd2035a34e10a0aa051fac14d63e2451ad77f0: Status 404 returned error can't find the container with id 137654b2a278ba1487854df4e3dd2035a34e10a0aa051fac14d63e2451ad77f0 Jul 16 13:46:45.200623 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.200580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:45.234748 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.234708 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxkwr"] Jul 16 13:46:45.239511 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.239486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.242560 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.242490 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 16 13:46:45.242678 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.242604 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vhscb\"" Jul 16 13:46:45.254462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.254434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxkwr"] Jul 16 13:46:45.303661 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.303616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.303860 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.303672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4n6\" (UniqueName: \"kubernetes.io/projected/7e77e136-2d2c-40a8-b152-7485f4a6339b-kube-api-access-2t4n6\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.404780 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.404745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.405000 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.404788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4n6\" (UniqueName: \"kubernetes.io/projected/7e77e136-2d2c-40a8-b152-7485f4a6339b-kube-api-access-2t4n6\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.405000 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:46:45.404931 2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 16 13:46:45.405121 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:46:45.405006 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs podName:7e77e136-2d2c-40a8-b152-7485f4a6339b nodeName:}" failed. No retries permitted until 2026-07-16 13:46:45.904983533 +0000 UTC m=+385.853100221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs") pod "model-serving-api-54d89bd5fc-kxkwr" (UID: "7e77e136-2d2c-40a8-b152-7485f4a6339b") : secret "model-serving-api-tls" not found Jul 16 13:46:45.421700 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.421660 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4n6\" (UniqueName: \"kubernetes.io/projected/7e77e136-2d2c-40a8-b152-7485f4a6339b-kube-api-access-2t4n6\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.547151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.547106 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:45.555223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.555196 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:45.558652 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.558630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.571285 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.571263 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 13:46:45.571419 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.571310 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 13:46:45.574192 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.574168 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-72hl6\"" Jul 16 13:46:45.574316 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.574306 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 13:46:45.574945 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.574925 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 13:46:45.586798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.586767 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:45.606801 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.606770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bv7\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.606814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.606843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.606951 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.606996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607237 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.607034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.607237 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.607098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.707893 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.707821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bv7\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.707893 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.707905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.707961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.708020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.708045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.708064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708142 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.708117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.708730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.708705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.710339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.710310 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.710339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.710338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.710471 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.710316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.710471 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.710369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.736447 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.736412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.737886 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.737846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bv7\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7\") pod \"istiod-openshift-gateway-7574459656-dlsvh\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.757793 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.757750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:45.869733 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.869630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:45.909989 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.909957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.914245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.914188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e77e136-2d2c-40a8-b152-7485f4a6339b-tls-certs\") pod \"model-serving-api-54d89bd5fc-kxkwr\" (UID: \"7e77e136-2d2c-40a8-b152-7485f4a6339b\") " pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:45.983051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:45.982867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:46.068953 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:46.068920 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:46.073320 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:46.072068 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d0482a_f1d3_4591_adca_a1cd4e79670d.slice/crio-1b1bea2e0f9ddd5fe5c9e3a579698884530f3b2f0d4659481ceb62aed676c2b8 WatchSource:0}: Error finding container 1b1bea2e0f9ddd5fe5c9e3a579698884530f3b2f0d4659481ceb62aed676c2b8: Status 404 returned error can't find the container with id 1b1bea2e0f9ddd5fe5c9e3a579698884530f3b2f0d4659481ceb62aed676c2b8 Jul 16 13:46:46.125854 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:46.125766 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" event={"ID":"bb228b98-3ec5-4802-a27e-8993fa0c68e9","Type":"ContainerStarted","Data":"137654b2a278ba1487854df4e3dd2035a34e10a0aa051fac14d63e2451ad77f0"} Jul 16 13:46:46.127366 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:46.127333 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" event={"ID":"06d0482a-f1d3-4591-adca-a1cd4e79670d","Type":"ContainerStarted","Data":"1b1bea2e0f9ddd5fe5c9e3a579698884530f3b2f0d4659481ceb62aed676c2b8"} Jul 16 13:46:46.152554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:46.152525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:46.161583 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:46.161556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:47.321573 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:47.321535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:48.383892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:48.383848 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kxkwr"] Jul 16 13:46:48.387572 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:46:48.387541 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e77e136_2d2c_40a8_b152_7485f4a6339b.slice/crio-dd150b5e0e9eeb4ed6432024df484215e7c6c4e5a17929733b3e82f9c5441b88 WatchSource:0}: Error finding container dd150b5e0e9eeb4ed6432024df484215e7c6c4e5a17929733b3e82f9c5441b88: Status 404 returned error can't find the container with id dd150b5e0e9eeb4ed6432024df484215e7c6c4e5a17929733b3e82f9c5441b88 Jul 16 13:46:49.148277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.148210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" event={"ID":"b7204850-f591-4438-8c51-ace01411191a","Type":"ContainerStarted","Data":"e9761ea1a31a35543b23bfe374d2c5e9d300ef827f9cf7e1655811b0567d053d"} Jul 16 13:46:49.150253 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.150215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" event={"ID":"bb228b98-3ec5-4802-a27e-8993fa0c68e9","Type":"ContainerStarted","Data":"35943f1dada1d7bd15bc5eca1bac27a3042603feb277cc7b4af267f9271fdc01"} Jul 16 13:46:49.150421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.150339 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:46:49.151737 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.151710 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" event={"ID":"7e77e136-2d2c-40a8-b152-7485f4a6339b","Type":"ContainerStarted","Data":"dd150b5e0e9eeb4ed6432024df484215e7c6c4e5a17929733b3e82f9c5441b88"} Jul 16 13:46:49.166607 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.166457 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b75db759-xwgcj" podStartSLOduration=2.034941518 podStartE2EDuration="6.16644008s" totalImagesPullingTime="4.131498562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:46:49.165925416 +0000 UTC m=+389.114042222" watchObservedRunningTime="2026-07-16 13:46:49.16644008 +0000 UTC m=+389.114556788" Jul 16 13:46:49.186511 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:49.185383 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" podStartSLOduration=2.066521743 podStartE2EDuration="5.185365868s" totalImagesPullingTime="3.118844125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:46:49.184910828 +0000 UTC m=+389.133027561" watchObservedRunningTime="2026-07-16 13:46:49.185365868 +0000 UTC m=+389.133482574" Jul 16 13:46:50.361768 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:50.361733 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:46:50.362140 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:50.361816 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:46:51.165287 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.165245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" event={"ID":"7e77e136-2d2c-40a8-b152-7485f4a6339b","Type":"ContainerStarted","Data":"143dd7ffc82b1ef55185d1921fa0a97391f9dedccc9d2e695067796ac7afc353"} Jul 16 13:46:51.165541 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.165308 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:46:51.167297 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.167235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" event={"ID":"06d0482a-f1d3-4591-adca-a1cd4e79670d","Type":"ContainerStarted","Data":"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02"} Jul 16 13:46:51.167963 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.167943 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:51.169530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.169507 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dlsvh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 13:46:51.169700 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.169672 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 13:46:51.189944 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.189887 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" podStartSLOduration=4.217817555 podStartE2EDuration="6.189857197s" totalImagesPullingTime="1.972039642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:46:51.187952627 +0000 UTC m=+391.136069352" watchObservedRunningTime="2026-07-16 13:46:51.189857197 +0000 UTC m=+391.137973904" Jul 16 13:46:51.194448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.194418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:51.212589 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.212532 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" podStartSLOduration=1.9272367639999999 podStartE2EDuration="6.212514355s" totalImagesPullingTime="4.285277591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:46:51.211909871 +0000 UTC m=+391.160026601" watchObservedRunningTime="2026-07-16 13:46:51.212514355 +0000 UTC m=+391.160631061" Jul 16 13:46:51.312505 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:51.312425 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:52.158885 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:52.158828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:52.172363 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:52.172336 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dlsvh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 13:46:52.172513 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:52.172384 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 13:46:52.485102 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:52.485012 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:52.939189 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:52.939150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:53.176601 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:53.176573 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:54.872664 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:54.872626 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:55.047715 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.047678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:55.266359 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.266323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:55.548609 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.548529 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:55.548758 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.548729 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" containerID="cri-o://26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02" gracePeriod=30 Jul 16 13:46:55.668701 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.668658 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:55.794536 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.794513 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:55.907503 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907394 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.907503 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907490 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907519 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907546 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4bv7\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907585 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.907678 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert\") pod \"06d0482a-f1d3-4591-adca-a1cd4e79670d\" (UID: \"06d0482a-f1d3-4591-adca-a1cd4e79670d\") " Jul 16 13:46:55.908127 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.908045 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:46:55.910043 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910016 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs" (OuterVolumeSpecName: "local-certs") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:46:55.910043 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910042 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7" (OuterVolumeSpecName: "kube-api-access-w4bv7") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "kube-api-access-w4bv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:46:55.910263 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910186 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:46:55.910263 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts" (OuterVolumeSpecName: "cacerts") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:46:55.910336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910266 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:46:55.910412 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:55.910398 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token" (OuterVolumeSpecName: "istio-token") pod "06d0482a-f1d3-4591-adca-a1cd4e79670d" (UID: "06d0482a-f1d3-4591-adca-a1cd4e79670d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:46:56.009317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009279 2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-kubeconfig\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009309 2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-cacerts\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009318 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bv7\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-kube-api-access-w4bv7\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009329 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009581 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009338 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-dns-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009581 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009347 2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d0482a-f1d3-4591-adca-a1cd4e79670d-local-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.009581 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.009357 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06d0482a-f1d3-4591-adca-a1cd4e79670d-istio-csr-ca-configmap\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:46:56.190934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.190851 2563 generic.go:356] "Generic (PLEG): container finished" podID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerID="26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02" exitCode=0 Jul 16 13:46:56.190934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.190904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" event={"ID":"06d0482a-f1d3-4591-adca-a1cd4e79670d","Type":"ContainerDied","Data":"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02"} Jul 16 13:46:56.191109 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.190958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" event={"ID":"06d0482a-f1d3-4591-adca-a1cd4e79670d","Type":"ContainerDied","Data":"1b1bea2e0f9ddd5fe5c9e3a579698884530f3b2f0d4659481ceb62aed676c2b8"} Jul 16 13:46:56.191109 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.190913 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh" Jul 16 13:46:56.191109 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.190981 2563 scope.go:117] "RemoveContainer" containerID="26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02" Jul 16 13:46:56.200648 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.200630 2563 scope.go:117] "RemoveContainer" containerID="26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02" Jul 16 13:46:56.200924 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:46:56.200901 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02\": container with ID starting with 26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02 not found: ID does not exist" containerID="26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02" Jul 16 13:46:56.201019 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.200930 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02"} err="failed to get container status \"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02\": rpc error: code = NotFound desc = could not find container \"26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02\": container with ID starting with 26aa256112ac95ca790820a1c17fe33b8801464c57400ff26493aa0c2aa21a02 not found: ID does not exist" Jul 16 13:46:56.215052 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.215015 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:56.223750 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.223724 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dlsvh"] Jul 16 13:46:56.261730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.261691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:46:56.474245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.474153 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" path="/var/lib/kubelet/pods/06d0482a-f1d3-4591-adca-a1cd4e79670d/volumes" Jul 16 13:46:56.476146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:46:56.476120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:00.160103 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:00.160075 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fctwd" Jul 16 13:47:02.176006 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:02.175976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kxkwr" Jul 16 13:47:10.270411 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:10.270374 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:10.338200 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:10.338164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:10.406941 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:10.406902 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:36.061520 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.061485 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6956dc7c-r5m4d"] Jul 16 13:47:36.062000 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.061829 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" Jul 16 13:47:36.062000 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.061841 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" Jul 16 13:47:36.062000 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.061923 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d0482a-f1d3-4591-adca-a1cd4e79670d" containerName="discovery" Jul 16 13:47:36.069715 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.069689 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.078140 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.078119 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6956dc7c-r5m4d"] Jul 16 13:47:36.131843 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.131805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:36.150130 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-oauth-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150267 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150267 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-trusted-ca-bundle\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150267 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150178 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-oauth-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150267 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-service-ca\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.150399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.150318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jjz\" (UniqueName: \"kubernetes.io/projected/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-kube-api-access-87jjz\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.193168 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.193133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:36.251401 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251365 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251401 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-trusted-ca-bundle\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251570 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-oauth-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251570 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251459 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251570 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-service-ca\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251570 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jjz\" (UniqueName: \"kubernetes.io/projected/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-kube-api-access-87jjz\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.251763 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.251596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-oauth-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.252329 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.252300 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-oauth-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.252507 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.252331 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.252507 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.252340 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-service-ca\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.252507 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.252432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-trusted-ca-bundle\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.254063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.254044 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-oauth-config\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.254156 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.254138 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-console-serving-cert\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.259069 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.259050 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jjz\" (UniqueName: \"kubernetes.io/projected/24ad6f90-c8fa-4a04-a549-ba75e7116c9c-kube-api-access-87jjz\") pod \"console-7d6956dc7c-r5m4d\" (UID: \"24ad6f90-c8fa-4a04-a549-ba75e7116c9c\") " pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.261543 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.261506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:36.318312 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.318220 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:36.380493 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.380456 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:36.515613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.515586 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6956dc7c-r5m4d"] Jul 16 13:47:36.517600 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:47:36.517574 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ad6f90_c8fa_4a04_a549_ba75e7116c9c.slice/crio-804e342dfb51fdc617c1cfea6cf296142ee7dd24ac889b5ebc08c85a8cc7062e WatchSource:0}: Error finding container 804e342dfb51fdc617c1cfea6cf296142ee7dd24ac889b5ebc08c85a8cc7062e: Status 404 returned error can't find the container with id 804e342dfb51fdc617c1cfea6cf296142ee7dd24ac889b5ebc08c85a8cc7062e Jul 16 13:47:36.867348 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:36.867246 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:37.382325 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:37.382287 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6956dc7c-r5m4d" event={"ID":"24ad6f90-c8fa-4a04-a549-ba75e7116c9c","Type":"ContainerStarted","Data":"af42ea055114a2c3d7fce28f4f23b10a2c59fb89eaef79ea75ea41ae68fdef74"} Jul 16 13:47:37.382819 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:37.382344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6956dc7c-r5m4d" event={"ID":"24ad6f90-c8fa-4a04-a549-ba75e7116c9c","Type":"ContainerStarted","Data":"804e342dfb51fdc617c1cfea6cf296142ee7dd24ac889b5ebc08c85a8cc7062e"} Jul 16 13:47:37.401228 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:37.401182 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6956dc7c-r5m4d" podStartSLOduration=1.4011649130000001 podStartE2EDuration="1.401164913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:47:37.400099597 +0000 UTC m=+437.348216304" watchObservedRunningTime="2026-07-16 13:47:37.401164913 +0000 UTC m=+437.349281620" Jul 16 13:47:44.879770 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:44.879536 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:44.974955 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:44.974918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:45.025262 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.025225 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:45.027743 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.027719 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:45.030366 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.030346 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 13:47:45.031242 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.031214 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-z2pdh\"" Jul 16 13:47:45.031242 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.031233 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 13:47:45.038933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.038908 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:45.089103 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.089068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:45.130768 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.130693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw276\" (UniqueName: \"kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276\") pod \"kuadrant-operator-catalog-2xvfv\" (UID: \"01c29ac8-6124-4ad6-9109-f4f8387e521a\") " pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:45.231518 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.231483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw276\" (UniqueName: \"kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276\") pod \"kuadrant-operator-catalog-2xvfv\" (UID: \"01c29ac8-6124-4ad6-9109-f4f8387e521a\") " pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:45.239691 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.239656 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw276\" (UniqueName: \"kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276\") pod \"kuadrant-operator-catalog-2xvfv\" (UID: \"01c29ac8-6124-4ad6-9109-f4f8387e521a\") " pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:45.339711 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.339663 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:45.379660 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.379629 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:45.475190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.475164 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:45.476857 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:47:45.476830 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c29ac8_6124_4ad6_9109_f4f8387e521a.slice/crio-4da622f74ed509a7a4c8a6cb0ade50819046a138e24acecb6f1937a32f518a41 WatchSource:0}: Error finding container 4da622f74ed509a7a4c8a6cb0ade50819046a138e24acecb6f1937a32f518a41: Status 404 returned error can't find the container with id 4da622f74ed509a7a4c8a6cb0ade50819046a138e24acecb6f1937a32f518a41 Jul 16 13:47:45.592772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.592736 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkdd4"] Jul 16 13:47:45.595377 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.595360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:45.607383 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.607360 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkdd4"] Jul 16 13:47:45.736579 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.736497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8cm\" (UniqueName: \"kubernetes.io/projected/e8ba6a14-303c-4ef2-8125-e0b1872b3a62-kube-api-access-fh8cm\") pod \"kuadrant-operator-catalog-gkdd4\" (UID: \"e8ba6a14-303c-4ef2-8125-e0b1872b3a62\") " pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:45.837579 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.837548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8cm\" (UniqueName: \"kubernetes.io/projected/e8ba6a14-303c-4ef2-8125-e0b1872b3a62-kube-api-access-fh8cm\") pod \"kuadrant-operator-catalog-gkdd4\" (UID: \"e8ba6a14-303c-4ef2-8125-e0b1872b3a62\") " pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:45.846772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.846746 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8cm\" (UniqueName: \"kubernetes.io/projected/e8ba6a14-303c-4ef2-8125-e0b1872b3a62-kube-api-access-fh8cm\") pod \"kuadrant-operator-catalog-gkdd4\" (UID: \"e8ba6a14-303c-4ef2-8125-e0b1872b3a62\") " pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:45.905018 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:45.904981 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:46.033135 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.033111 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gkdd4"] Jul 16 13:47:46.035129 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:47:46.035097 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ba6a14_303c_4ef2_8125_e0b1872b3a62.slice/crio-2b952530bb379fab56ab148ef3270a33e054a64f4c455e0ed050d1f2e961a136 WatchSource:0}: Error finding container 2b952530bb379fab56ab148ef3270a33e054a64f4c455e0ed050d1f2e961a136: Status 404 returned error can't find the container with id 2b952530bb379fab56ab148ef3270a33e054a64f4c455e0ed050d1f2e961a136 Jul 16 13:47:46.381252 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.381172 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:46.381421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.381263 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:46.386944 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.386907 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:46.426145 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.426109 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" event={"ID":"01c29ac8-6124-4ad6-9109-f4f8387e521a","Type":"ContainerStarted","Data":"4da622f74ed509a7a4c8a6cb0ade50819046a138e24acecb6f1937a32f518a41"} Jul 16 13:47:46.427447 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.427419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" event={"ID":"e8ba6a14-303c-4ef2-8125-e0b1872b3a62","Type":"ContainerStarted","Data":"2b952530bb379fab56ab148ef3270a33e054a64f4c455e0ed050d1f2e961a136"} Jul 16 13:47:46.432706 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.432683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6956dc7c-r5m4d" Jul 16 13:47:46.486794 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.486761 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:47:46.603670 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.603634 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:46.703735 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.703660 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:46.770851 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:46.770809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:48.439320 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.439285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" event={"ID":"01c29ac8-6124-4ad6-9109-f4f8387e521a","Type":"ContainerStarted","Data":"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d"} Jul 16 13:47:48.439785 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.439345 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" podUID="01c29ac8-6124-4ad6-9109-f4f8387e521a" containerName="registry-server" containerID="cri-o://2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d" gracePeriod=2 Jul 16 13:47:48.440842 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.440816 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" event={"ID":"e8ba6a14-303c-4ef2-8125-e0b1872b3a62","Type":"ContainerStarted","Data":"4cdf87e1c09867e4f2f7f9a81e7acbc781fedbde8703ae06d56d09313adc7469"} Jul 16 13:47:48.456705 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.456656 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" podStartSLOduration=0.972332682 podStartE2EDuration="3.456642105s" totalImagesPullingTime="2.484309423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:47:48.454680944 +0000 UTC m=+448.402797655" watchObservedRunningTime="2026-07-16 13:47:48.456642105 +0000 UTC m=+448.404758811" Jul 16 13:47:48.471348 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.471296 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" podStartSLOduration=1.5422003210000002 podStartE2EDuration="3.471279671s" totalImagesPullingTime="1.92907935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:47:48.471019506 +0000 UTC m=+448.419136269" watchObservedRunningTime="2026-07-16 13:47:48.471279671 +0000 UTC m=+448.419396378" Jul 16 13:47:48.683241 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.683188 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:48.764172 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.764152 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw276\" (UniqueName: \"kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276\") pod \"01c29ac8-6124-4ad6-9109-f4f8387e521a\" (UID: \"01c29ac8-6124-4ad6-9109-f4f8387e521a\") " Jul 16 13:47:48.766284 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.766261 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276" (OuterVolumeSpecName: "kube-api-access-kw276") pod "01c29ac8-6124-4ad6-9109-f4f8387e521a" (UID: "01c29ac8-6124-4ad6-9109-f4f8387e521a"). InnerVolumeSpecName "kube-api-access-kw276". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:47:48.865773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:48.865732 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw276\" (UniqueName: \"kubernetes.io/projected/01c29ac8-6124-4ad6-9109-f4f8387e521a-kube-api-access-kw276\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:47:49.446908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.446856 2563 generic.go:356] "Generic (PLEG): container finished" podID="01c29ac8-6124-4ad6-9109-f4f8387e521a" containerID="2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d" exitCode=0 Jul 16 13:47:49.447351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.446912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" event={"ID":"01c29ac8-6124-4ad6-9109-f4f8387e521a","Type":"ContainerDied","Data":"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d"} Jul 16 13:47:49.447351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.446921 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" Jul 16 13:47:49.447351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.446988 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2xvfv" event={"ID":"01c29ac8-6124-4ad6-9109-f4f8387e521a","Type":"ContainerDied","Data":"4da622f74ed509a7a4c8a6cb0ade50819046a138e24acecb6f1937a32f518a41"} Jul 16 13:47:49.447351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.447001 2563 scope.go:117] "RemoveContainer" containerID="2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d" Jul 16 13:47:49.456179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.456159 2563 scope.go:117] "RemoveContainer" containerID="2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d" Jul 16 13:47:49.456430 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:47:49.456411 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d\": container with ID starting with 2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d not found: ID does not exist" containerID="2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d" Jul 16 13:47:49.456478 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.456439 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d"} err="failed to get container status \"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d\": rpc error: code = NotFound desc = could not find container \"2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d\": container with ID starting with 2b0b283af4dd49d80b6d6049b0cc0882eaff7f7c84f2c395aea5236f6c54724d not found: ID does not exist" Jul 16 13:47:49.467913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.467870 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:49.471537 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:49.471515 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2xvfv"] Jul 16 13:47:50.473324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:50.473295 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c29ac8-6124-4ad6-9109-f4f8387e521a" path="/var/lib/kubelet/pods/01c29ac8-6124-4ad6-9109-f4f8387e521a/volumes" Jul 16 13:47:55.905732 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:55.905684 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:55.906231 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:55.905770 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:55.927987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:55.927958 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:56.502125 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:56.502098 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gkdd4" Jul 16 13:47:59.898216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:59.898182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:47:59.999260 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:47:59.999224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:00.609558 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.609526 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n"] Jul 16 13:48:00.609908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.609895 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c29ac8-6124-4ad6-9109-f4f8387e521a" containerName="registry-server" Jul 16 13:48:00.609961 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.609910 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c29ac8-6124-4ad6-9109-f4f8387e521a" containerName="registry-server" Jul 16 13:48:00.610002 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.609982 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c29ac8-6124-4ad6-9109-f4f8387e521a" containerName="registry-server" Jul 16 13:48:00.614501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.614485 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.616782 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.616761 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h2wwc\"" Jul 16 13:48:00.624146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.624124 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n"] Jul 16 13:48:00.770496 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.770455 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.770670 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.770509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.770670 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.770578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lmp\" (UniqueName: \"kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.872079 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.871990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.872079 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.872066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lmp\" (UniqueName: \"kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.872259 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.872142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.872404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.872386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.872493 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.872476 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.880501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.880479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lmp\" (UniqueName: \"kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:00.924525 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:00.924489 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:01.003496 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.003465 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq"] Jul 16 13:48:01.008193 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.008169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.017678 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.017655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq"] Jul 16 13:48:01.066489 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.066450 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n"] Jul 16 13:48:01.068663 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:01.068635 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d99b1fa_1f2f_4346_8918_e74c2208c918.slice/crio-5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c WatchSource:0}: Error finding container 5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c: Status 404 returned error can't find the container with id 5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c Jul 16 13:48:01.179289 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.179246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5zf\" (UniqueName: \"kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.179434 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.179327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.179481 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.179425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.280575 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.280532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5zf\" (UniqueName: \"kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.280752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.280588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.280752 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.280626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.281041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.281024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.281095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.281063 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.289087 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.289069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5zf\" (UniqueName: \"kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.321172 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.321133 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:01.416163 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.416130 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv"] Jul 16 13:48:01.421068 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.421036 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.434054 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.434024 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv"] Jul 16 13:48:01.465079 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.465053 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq"] Jul 16 13:48:01.467734 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:01.467699 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd66ebd_8e44_4b6c_afc1_9ae5fedaf924.slice/crio-1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d WatchSource:0}: Error finding container 1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d: Status 404 returned error can't find the container with id 1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d Jul 16 13:48:01.502669 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.502634 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerStarted","Data":"1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d"} Jul 16 13:48:01.504129 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.504100 2563 generic.go:356] "Generic (PLEG): container finished" podID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerID="7ad2f3d331f3f987a1a0e3f3431181e6b75e1a0a5a1f8b8ee5f10e569d45503c" exitCode=0 Jul 16 13:48:01.504256 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.504189 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" event={"ID":"1d99b1fa-1f2f-4346-8918-e74c2208c918","Type":"ContainerDied","Data":"7ad2f3d331f3f987a1a0e3f3431181e6b75e1a0a5a1f8b8ee5f10e569d45503c"} Jul 16 13:48:01.504256 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.504246 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" event={"ID":"1d99b1fa-1f2f-4346-8918-e74c2208c918","Type":"ContainerStarted","Data":"5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c"} Jul 16 13:48:01.584494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.584323 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.584494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.584377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmp2w\" (UniqueName: \"kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.584622 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.584491 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.685947 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.685820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.685947 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.685859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmp2w\" (UniqueName: \"kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.685947 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.685929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.686291 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.686268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.686353 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.686272 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.695146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.695121 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmp2w\" (UniqueName: \"kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.733662 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.733628 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:01.810332 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.810302 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7"] Jul 16 13:48:01.815044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.815027 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:01.825031 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.825006 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7"] Jul 16 13:48:01.869981 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.869957 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv"] Jul 16 13:48:01.871804 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:01.871766 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce7cafc_c833_4dd7_b566_4d7620d2eb2a.slice/crio-2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5 WatchSource:0}: Error finding container 2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5: Status 404 returned error can't find the container with id 2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5 Jul 16 13:48:01.988455 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.988423 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:01.988804 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.988462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:01.988804 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:01.988578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gsh\" (UniqueName: \"kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.089935 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.089903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.089935 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.089942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.090198 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.089998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gsh\" (UniqueName: \"kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.090390 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.090369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.090456 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.090427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.099721 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.099691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gsh\" (UniqueName: \"kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.127837 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.127803 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:02.270863 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.270840 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7"] Jul 16 13:48:02.276645 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:02.276607 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15be7cdd_1996_4b4e_843f_b2874979801f.slice/crio-650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b WatchSource:0}: Error finding container 650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b: Status 404 returned error can't find the container with id 650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b Jul 16 13:48:02.511297 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.511200 2563 generic.go:356] "Generic (PLEG): container finished" podID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerID="b215038174f6805274b0bd4e3d5a67d43799b78532239c3cc5c5186b2b2e3978" exitCode=0 Jul 16 13:48:02.511297 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.511270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerDied","Data":"b215038174f6805274b0bd4e3d5a67d43799b78532239c3cc5c5186b2b2e3978"} Jul 16 13:48:02.512810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.512786 2563 generic.go:356] "Generic (PLEG): container finished" podID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerID="52d3538be4a8007a68d721e1ff8ea9afe27eb2665ee6f463450bf28da3107738" exitCode=0 Jul 16 13:48:02.512925 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.512899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" event={"ID":"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a","Type":"ContainerDied","Data":"52d3538be4a8007a68d721e1ff8ea9afe27eb2665ee6f463450bf28da3107738"} Jul 16 13:48:02.513012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.512950 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" event={"ID":"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a","Type":"ContainerStarted","Data":"2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5"} Jul 16 13:48:02.514705 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.514677 2563 generic.go:356] "Generic (PLEG): container finished" podID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerID="c5c37439e0f839edbac5125359bf4c9af7fc8d4e8f4e7d3015818121c921844f" exitCode=0 Jul 16 13:48:02.514867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.514736 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" event={"ID":"1d99b1fa-1f2f-4346-8918-e74c2208c918","Type":"ContainerDied","Data":"c5c37439e0f839edbac5125359bf4c9af7fc8d4e8f4e7d3015818121c921844f"} Jul 16 13:48:02.516401 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.516377 2563 generic.go:356] "Generic (PLEG): container finished" podID="15be7cdd-1996-4b4e-843f-b2874979801f" containerID="e7acbda7629f4b3983a9433b159725c18db637b626e7fc29f30b3868ed163a4c" exitCode=0 Jul 16 13:48:02.516490 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.516399 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" event={"ID":"15be7cdd-1996-4b4e-843f-b2874979801f","Type":"ContainerDied","Data":"e7acbda7629f4b3983a9433b159725c18db637b626e7fc29f30b3868ed163a4c"} Jul 16 13:48:02.516490 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:02.516439 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" event={"ID":"15be7cdd-1996-4b4e-843f-b2874979801f","Type":"ContainerStarted","Data":"650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b"} Jul 16 13:48:03.525090 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:03.525058 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerStarted","Data":"845c90ebc5068adbffe7e95e21e80193fd03755f5ed2f907bd001faa61ed2d66"} Jul 16 13:48:03.527012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:03.526986 2563 generic.go:356] "Generic (PLEG): container finished" podID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerID="b9753e5960afc1b35f9b5a858a4ec1dc89fcaa4b7316f61c0204c14eb0fdf7ee" exitCode=0 Jul 16 13:48:03.527154 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:03.527070 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" event={"ID":"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a","Type":"ContainerDied","Data":"b9753e5960afc1b35f9b5a858a4ec1dc89fcaa4b7316f61c0204c14eb0fdf7ee"} Jul 16 13:48:03.529517 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:03.529493 2563 generic.go:356] "Generic (PLEG): container finished" podID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerID="12ad28163f54950dbbe500a6d5b84af8528894c46691994b23d7cb38ad34b989" exitCode=0 Jul 16 13:48:03.529615 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:03.529558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" event={"ID":"1d99b1fa-1f2f-4346-8918-e74c2208c918","Type":"ContainerDied","Data":"12ad28163f54950dbbe500a6d5b84af8528894c46691994b23d7cb38ad34b989"} Jul 16 13:48:04.536462 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.536426 2563 generic.go:356] "Generic (PLEG): container finished" podID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerID="845c90ebc5068adbffe7e95e21e80193fd03755f5ed2f907bd001faa61ed2d66" exitCode=0 Jul 16 13:48:04.536969 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.536514 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerDied","Data":"845c90ebc5068adbffe7e95e21e80193fd03755f5ed2f907bd001faa61ed2d66"} Jul 16 13:48:04.538713 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.538692 2563 generic.go:356] "Generic (PLEG): container finished" podID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerID="47dedf0b814f22e89f68dfc3482800eab9c7d04233c52c98982497161c3b6e12" exitCode=0 Jul 16 13:48:04.538801 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.538773 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" event={"ID":"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a","Type":"ContainerDied","Data":"47dedf0b814f22e89f68dfc3482800eab9c7d04233c52c98982497161c3b6e12"} Jul 16 13:48:04.540755 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.540734 2563 generic.go:356] "Generic (PLEG): container finished" podID="15be7cdd-1996-4b4e-843f-b2874979801f" containerID="993a6f4c94abae093bbc042c1ac776c41ad5d309ccb76958867f469d9a8bc054" exitCode=0 Jul 16 13:48:04.540848 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.540818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" event={"ID":"15be7cdd-1996-4b4e-843f-b2874979801f","Type":"ContainerDied","Data":"993a6f4c94abae093bbc042c1ac776c41ad5d309ccb76958867f469d9a8bc054"} Jul 16 13:48:04.693118 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.693092 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:04.812353 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.812267 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2lmp\" (UniqueName: \"kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp\") pod \"1d99b1fa-1f2f-4346-8918-e74c2208c918\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " Jul 16 13:48:04.812353 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.812328 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util\") pod \"1d99b1fa-1f2f-4346-8918-e74c2208c918\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " Jul 16 13:48:04.812541 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.812410 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle\") pod \"1d99b1fa-1f2f-4346-8918-e74c2208c918\" (UID: \"1d99b1fa-1f2f-4346-8918-e74c2208c918\") " Jul 16 13:48:04.812972 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.812936 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle" (OuterVolumeSpecName: "bundle") pod "1d99b1fa-1f2f-4346-8918-e74c2208c918" (UID: "1d99b1fa-1f2f-4346-8918-e74c2208c918"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:04.814369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.814343 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp" (OuterVolumeSpecName: "kube-api-access-h2lmp") pod "1d99b1fa-1f2f-4346-8918-e74c2208c918" (UID: "1d99b1fa-1f2f-4346-8918-e74c2208c918"). InnerVolumeSpecName "kube-api-access-h2lmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:48:04.817769 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.817749 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util" (OuterVolumeSpecName: "util") pod "1d99b1fa-1f2f-4346-8918-e74c2208c918" (UID: "1d99b1fa-1f2f-4346-8918-e74c2208c918"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:04.913182 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.913148 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:04.913182 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.913178 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lmp\" (UniqueName: \"kubernetes.io/projected/1d99b1fa-1f2f-4346-8918-e74c2208c918-kube-api-access-h2lmp\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:04.913182 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:04.913188 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d99b1fa-1f2f-4346-8918-e74c2208c918-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:05.547942 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.547900 2563 generic.go:356] "Generic (PLEG): container finished" podID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerID="923fd5de50fccd6d5eb04d959c6a40f914e938f362e07980f4a83ec82805a686" exitCode=0 Jul 16 13:48:05.548368 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.547979 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerDied","Data":"923fd5de50fccd6d5eb04d959c6a40f914e938f362e07980f4a83ec82805a686"} Jul 16 13:48:05.549832 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.549811 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" Jul 16 13:48:05.549950 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.549831 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n" event={"ID":"1d99b1fa-1f2f-4346-8918-e74c2208c918","Type":"ContainerDied","Data":"5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c"} Jul 16 13:48:05.549950 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.549887 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5417917b4e3d3c4647605cdda6e70dd1b993b944fa8ed52092be5880e8baec0c" Jul 16 13:48:05.552005 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.551982 2563 generic.go:356] "Generic (PLEG): container finished" podID="15be7cdd-1996-4b4e-843f-b2874979801f" containerID="5fd08b3c031f3f6d91ad08d01e6e40e8fa56ffde2ca67b3e8496d35317e2e83a" exitCode=0 Jul 16 13:48:05.552092 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.552061 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" event={"ID":"15be7cdd-1996-4b4e-843f-b2874979801f","Type":"ContainerDied","Data":"5fd08b3c031f3f6d91ad08d01e6e40e8fa56ffde2ca67b3e8496d35317e2e83a"} Jul 16 13:48:05.686064 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.686040 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:05.821041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.820945 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle\") pod \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " Jul 16 13:48:05.821041 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.820995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmp2w\" (UniqueName: \"kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w\") pod \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " Jul 16 13:48:05.821275 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.821052 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util\") pod \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\" (UID: \"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a\") " Jul 16 13:48:05.821567 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.821535 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle" (OuterVolumeSpecName: "bundle") pod "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" (UID: "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:05.823252 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.823227 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w" (OuterVolumeSpecName: "kube-api-access-tmp2w") pod "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" (UID: "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a"). InnerVolumeSpecName "kube-api-access-tmp2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:48:05.826775 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.826740 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util" (OuterVolumeSpecName: "util") pod "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" (UID: "5ce7cafc-c833-4dd7-b566-4d7620d2eb2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:05.921771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.921739 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:05.921771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.921769 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmp2w\" (UniqueName: \"kubernetes.io/projected/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-kube-api-access-tmp2w\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:05.921771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:05.921778 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ce7cafc-c833-4dd7-b566-4d7620d2eb2a-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.559037 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.559001 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" Jul 16 13:48:06.559413 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.559001 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv" event={"ID":"5ce7cafc-c833-4dd7-b566-4d7620d2eb2a","Type":"ContainerDied","Data":"2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5"} Jul 16 13:48:06.559413 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.559120 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5419090d6c592ab2e212555de06942bda5355b599dc94878143273844e15f5" Jul 16 13:48:06.697574 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.697547 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:06.725193 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.725170 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:06.829078 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829032 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle\") pod \"15be7cdd-1996-4b4e-843f-b2874979801f\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " Jul 16 13:48:06.829078 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829082 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util\") pod \"15be7cdd-1996-4b4e-843f-b2874979801f\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " Jul 16 13:48:06.829339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829109 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb5zf\" (UniqueName: \"kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf\") pod \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " Jul 16 13:48:06.829339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829132 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9gsh\" (UniqueName: \"kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh\") pod \"15be7cdd-1996-4b4e-843f-b2874979801f\" (UID: \"15be7cdd-1996-4b4e-843f-b2874979801f\") " Jul 16 13:48:06.829339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829236 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util\") pod \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " Jul 16 13:48:06.829339 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829267 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle\") pod \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\" (UID: \"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924\") " Jul 16 13:48:06.829800 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829765 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle" (OuterVolumeSpecName: "bundle") pod "15be7cdd-1996-4b4e-843f-b2874979801f" (UID: "15be7cdd-1996-4b4e-843f-b2874979801f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:06.829940 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.829795 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle" (OuterVolumeSpecName: "bundle") pod "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" (UID: "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:06.831257 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.831233 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh" (OuterVolumeSpecName: "kube-api-access-z9gsh") pod "15be7cdd-1996-4b4e-843f-b2874979801f" (UID: "15be7cdd-1996-4b4e-843f-b2874979801f"). InnerVolumeSpecName "kube-api-access-z9gsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:48:06.831406 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.831392 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf" (OuterVolumeSpecName: "kube-api-access-jb5zf") pod "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" (UID: "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924"). InnerVolumeSpecName "kube-api-access-jb5zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:48:06.838396 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.838368 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util" (OuterVolumeSpecName: "util") pod "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" (UID: "edd66ebd-8e44-4b6c-afc1-9ae5fedaf924"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:06.838672 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.838653 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util" (OuterVolumeSpecName: "util") pod "15be7cdd-1996-4b4e-843f-b2874979801f" (UID: "15be7cdd-1996-4b4e-843f-b2874979801f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:48:06.930773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930737 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.930773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930766 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15be7cdd-1996-4b4e-843f-b2874979801f-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.930773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930778 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb5zf\" (UniqueName: \"kubernetes.io/projected/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-kube-api-access-jb5zf\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.931008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930787 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gsh\" (UniqueName: \"kubernetes.io/projected/15be7cdd-1996-4b4e-843f-b2874979801f-kube-api-access-z9gsh\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.931008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930796 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-util\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:06.931008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:06.930804 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd66ebd-8e44-4b6c-afc1-9ae5fedaf924-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:07.565927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.565896 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" Jul 16 13:48:07.565927 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.565908 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq" event={"ID":"edd66ebd-8e44-4b6c-afc1-9ae5fedaf924","Type":"ContainerDied","Data":"1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d"} Jul 16 13:48:07.566375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.565956 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1870a755a6d790bd087da658e0e026886adb320976c09ebfbbd8c6eb48802a8d" Jul 16 13:48:07.567828 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.567804 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" event={"ID":"15be7cdd-1996-4b4e-843f-b2874979801f","Type":"ContainerDied","Data":"650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b"} Jul 16 13:48:07.567828 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.567820 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7" Jul 16 13:48:07.568100 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:07.567838 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650462762c40acedffcb99f729a9a77399f18e9bd0ee00c9354159e6b9648c6b" Jul 16 13:48:11.510333 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.510297 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c757b96fc-l2rqx" podUID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" containerName="console" containerID="cri-o://5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3" gracePeriod=15 Jul 16 13:48:11.763387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.763326 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:48:11.770271 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770253 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770298 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770335 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770445 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770355 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770445 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770398 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770445 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770420 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770445 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770442 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert\") pod \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\" (UID: \"fc195628-c9ef-47a2-a9d0-7213d52d57f6\") " Jul 16 13:48:11.770730 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:48:11.770907 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770870 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:48:11.770987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.770909 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config" (OuterVolumeSpecName: "console-config") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:48:11.771060 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.771000 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:48:11.772492 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.772461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:48:11.772568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.772501 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:48:11.772614 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.772566 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87" (OuterVolumeSpecName: "kube-api-access-cbl87") pod "fc195628-c9ef-47a2-a9d0-7213d52d57f6" (UID: "fc195628-c9ef-47a2-a9d0-7213d52d57f6"). InnerVolumeSpecName "kube-api-access-cbl87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:48:11.871628 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871591 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-service-ca\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871628 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871626 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl87\" (UniqueName: \"kubernetes.io/projected/fc195628-c9ef-47a2-a9d0-7213d52d57f6-kube-api-access-cbl87\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871640 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871652 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871663 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-oauth-serving-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871673 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc195628-c9ef-47a2-a9d0-7213d52d57f6-trusted-ca-bundle\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:11.871844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:11.871684 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc195628-c9ef-47a2-a9d0-7213d52d57f6-console-oauth-config\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:48:12.592755 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.592716 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" containerID="5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3" exitCode=0 Jul 16 13:48:12.593171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.592764 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c757b96fc-l2rqx" Jul 16 13:48:12.593171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.592806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c757b96fc-l2rqx" event={"ID":"fc195628-c9ef-47a2-a9d0-7213d52d57f6","Type":"ContainerDied","Data":"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3"} Jul 16 13:48:12.593171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.592862 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c757b96fc-l2rqx" event={"ID":"fc195628-c9ef-47a2-a9d0-7213d52d57f6","Type":"ContainerDied","Data":"ce80551c773fb9f4584ea72e2e0ef6676ba27b7fa5faa50d41f60356a12b938b"} Jul 16 13:48:12.593171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.592912 2563 scope.go:117] "RemoveContainer" containerID="5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3" Jul 16 13:48:12.601932 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.601908 2563 scope.go:117] "RemoveContainer" containerID="5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3" Jul 16 13:48:12.602197 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:48:12.602175 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3\": container with ID starting with 5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3 not found: ID does not exist" containerID="5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3" Jul 16 13:48:12.602259 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.602206 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3"} err="failed to get container status \"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3\": rpc error: code = NotFound desc = could not find container \"5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3\": container with ID starting with 5d66a1afb266726b68f3da963d167f387bd89d0b836e036584829f80c61cc9b3 not found: ID does not exist" Jul 16 13:48:12.609813 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.609782 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:48:12.613821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:12.613791 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c757b96fc-l2rqx"] Jul 16 13:48:13.357299 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:13.357268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:13.954290 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:13.954259 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:14.474926 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:14.474895 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" path="/var/lib/kubelet/pods/fc195628-c9ef-47a2-a9d0-7213d52d57f6/volumes" Jul 16 13:48:14.990215 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:14.990183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:17.982215 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982178 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dljr"] Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982519 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982529 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982542 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="extract" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982547 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="extract" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982558 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982563 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982569 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="extract" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982573 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="extract" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982582 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="util" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982586 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="util" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982592 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982598 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982604 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982608 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="pull" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982616 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="util" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982622 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="util" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982632 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="extract" Jul 16 13:48:17.982632 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982638 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982647 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="util" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982652 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="util" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982657 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982661 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982670 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="util" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982674 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="util" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982680 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" containerName="console" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982685 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" containerName="console" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982734 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15be7cdd-1996-4b4e-843f-b2874979801f" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982740 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc195628-c9ef-47a2-a9d0-7213d52d57f6" containerName="console" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982747 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d99b1fa-1f2f-4346-8918-e74c2208c918" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982753 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce7cafc-c833-4dd7-b566-4d7620d2eb2a" containerName="extract" Jul 16 13:48:17.983216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.982760 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd66ebd-8e44-4b6c-afc1-9ae5fedaf924" containerName="extract" Jul 16 13:48:17.985829 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.985813 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:17.988216 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.988197 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bsnnt\"" Jul 16 13:48:17.997899 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:17.997860 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dljr"] Jul 16 13:48:18.023241 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.023206 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fn8x\" (UniqueName: \"kubernetes.io/projected/4ebf3195-b824-47e9-8b24-719b02824133-kube-api-access-5fn8x\") pod \"authorino-operator-75cc8d5cb7-7dljr\" (UID: \"4ebf3195-b824-47e9-8b24-719b02824133\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:18.124121 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.124086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fn8x\" (UniqueName: \"kubernetes.io/projected/4ebf3195-b824-47e9-8b24-719b02824133-kube-api-access-5fn8x\") pod \"authorino-operator-75cc8d5cb7-7dljr\" (UID: \"4ebf3195-b824-47e9-8b24-719b02824133\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:18.133379 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.133345 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fn8x\" (UniqueName: \"kubernetes.io/projected/4ebf3195-b824-47e9-8b24-719b02824133-kube-api-access-5fn8x\") pod \"authorino-operator-75cc8d5cb7-7dljr\" (UID: \"4ebf3195-b824-47e9-8b24-719b02824133\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:18.296044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.296002 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:18.431592 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.431565 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7dljr"] Jul 16 13:48:18.434536 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:18.434498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebf3195_b824_47e9_8b24_719b02824133.slice/crio-1d0306e7ace6982f8325f55efd9ba45f94861aebbb104994be0501047177fdc6 WatchSource:0}: Error finding container 1d0306e7ace6982f8325f55efd9ba45f94861aebbb104994be0501047177fdc6: Status 404 returned error can't find the container with id 1d0306e7ace6982f8325f55efd9ba45f94861aebbb104994be0501047177fdc6 Jul 16 13:48:18.630577 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:18.630491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" event={"ID":"4ebf3195-b824-47e9-8b24-719b02824133","Type":"ContainerStarted","Data":"1d0306e7ace6982f8325f55efd9ba45f94861aebbb104994be0501047177fdc6"} Jul 16 13:48:19.969511 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:19.969481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:21.650290 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:21.650250 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" event={"ID":"4ebf3195-b824-47e9-8b24-719b02824133","Type":"ContainerStarted","Data":"f402bbbd4404579da4eab3f95da48b302bb64899b2f32307f491fdee27694bb5"} Jul 16 13:48:21.650719 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:21.650354 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:21.674165 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:21.674110 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" podStartSLOduration=2.458067346 podStartE2EDuration="4.674096708s" totalImagesPullingTime="2.216029362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:48:21.672361606 +0000 UTC m=+481.620478312" watchObservedRunningTime="2026-07-16 13:48:21.674096708 +0000 UTC m=+481.622213415" Jul 16 13:48:22.474438 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.474408 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg"] Jul 16 13:48:22.478008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.477984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:22.480659 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.480639 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wf26d\"" Jul 16 13:48:22.490052 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.490031 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg"] Jul 16 13:48:22.562141 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.562106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn77m\" (UniqueName: \"kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m\") pod \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" (UID: \"53fa5559-14d9-49a1-9986-502214df261f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:22.663325 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.663296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn77m\" (UniqueName: \"kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m\") pod \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" (UID: \"53fa5559-14d9-49a1-9986-502214df261f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:22.676217 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.676181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn77m\" (UniqueName: \"kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m\") pod \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" (UID: \"53fa5559-14d9-49a1-9986-502214df261f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:22.788384 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.788347 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:22.920622 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:22.920595 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg"] Jul 16 13:48:22.922683 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:22.922654 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fa5559_14d9_49a1_9986_502214df261f.slice/crio-743bef5a66ac8a41e139e34228bd061d5d771e134616f6dcfcab39efd662a28b WatchSource:0}: Error finding container 743bef5a66ac8a41e139e34228bd061d5d771e134616f6dcfcab39efd662a28b: Status 404 returned error can't find the container with id 743bef5a66ac8a41e139e34228bd061d5d771e134616f6dcfcab39efd662a28b Jul 16 13:48:23.664181 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:23.664146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" event={"ID":"53fa5559-14d9-49a1-9986-502214df261f","Type":"ContainerStarted","Data":"743bef5a66ac8a41e139e34228bd061d5d771e134616f6dcfcab39efd662a28b"} Jul 16 13:48:24.168320 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:24.168279 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:24.670501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:24.670474 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" event={"ID":"53fa5559-14d9-49a1-9986-502214df261f","Type":"ContainerStarted","Data":"0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb"} Jul 16 13:48:24.670935 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:24.670590 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:24.696577 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:24.696528 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" podStartSLOduration=1.030454231 podStartE2EDuration="2.696513537s" totalImagesPullingTime="1.666059306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:48:24.695364998 +0000 UTC m=+484.643481704" watchObservedRunningTime="2026-07-16 13:48:24.696513537 +0000 UTC m=+484.644630243" Jul 16 13:48:26.586460 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.586426 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h"] Jul 16 13:48:26.589972 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.589952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:26.593075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.593034 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 13:48:26.593075 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.593051 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mwdrv\"" Jul 16 13:48:26.607989 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.607963 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h"] Jul 16 13:48:26.700255 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.700219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brcv\" (UniqueName: \"kubernetes.io/projected/adc7174e-48ed-485e-b686-58461c6f7c9c-kube-api-access-6brcv\") pod \"dns-operator-controller-manager-864dd78c4c-gzp4h\" (UID: \"adc7174e-48ed-485e-b686-58461c6f7c9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:26.801163 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.801128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brcv\" (UniqueName: \"kubernetes.io/projected/adc7174e-48ed-485e-b686-58461c6f7c9c-kube-api-access-6brcv\") pod \"dns-operator-controller-manager-864dd78c4c-gzp4h\" (UID: \"adc7174e-48ed-485e-b686-58461c6f7c9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:26.814359 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.814325 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brcv\" (UniqueName: \"kubernetes.io/projected/adc7174e-48ed-485e-b686-58461c6f7c9c-kube-api-access-6brcv\") pod \"dns-operator-controller-manager-864dd78c4c-gzp4h\" (UID: \"adc7174e-48ed-485e-b686-58461c6f7c9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:26.900521 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:26.900441 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:27.035012 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:27.034986 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h"] Jul 16 13:48:27.036941 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:27.036915 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc7174e_48ed_485e_b686_58461c6f7c9c.slice/crio-8302389aea42d9ed8272385a0454112a8c5f2b6bde773278ec4600feca263710 WatchSource:0}: Error finding container 8302389aea42d9ed8272385a0454112a8c5f2b6bde773278ec4600feca263710: Status 404 returned error can't find the container with id 8302389aea42d9ed8272385a0454112a8c5f2b6bde773278ec4600feca263710 Jul 16 13:48:27.164104 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:27.164010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:27.687246 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:27.687213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" event={"ID":"adc7174e-48ed-485e-b686-58461c6f7c9c","Type":"ContainerStarted","Data":"8302389aea42d9ed8272385a0454112a8c5f2b6bde773278ec4600feca263710"} Jul 16 13:48:29.428447 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:29.428404 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:29.698500 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:29.698461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" event={"ID":"adc7174e-48ed-485e-b686-58461c6f7c9c","Type":"ContainerStarted","Data":"6b16e02cab2d7906d64a90527a070d9ec90f06a0207d9a88ba82ce52df2eb845"} Jul 16 13:48:29.698864 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:29.698547 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:29.736110 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:29.736056 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" podStartSLOduration=1.5866132560000001 podStartE2EDuration="3.73603844s" totalImagesPullingTime="2.149425184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:48:29.733338156 +0000 UTC m=+489.681454862" watchObservedRunningTime="2026-07-16 13:48:29.73603844 +0000 UTC m=+489.684155147" Jul 16 13:48:31.085418 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.085374 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:31.653008 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.652971 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm"] Jul 16 13:48:31.656571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.656553 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.659691 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.659673 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5pqhk\"" Jul 16 13:48:31.684452 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.684419 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm"] Jul 16 13:48:31.742588 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.742549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.742757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.742596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dcv\" (UniqueName: \"kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.843401 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.843354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.843401 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.843408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dcv\" (UniqueName: \"kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.843845 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.843819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.857107 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.857082 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dcv\" (UniqueName: \"kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:31.967222 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:31.967132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:32.102929 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:32.102904 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm"] Jul 16 13:48:32.104568 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:32.104533 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed29fd2_7268_429d_acbb_91979daf794f.slice/crio-f1c0a8ca9d2a162d90c9bf2e87eeb23c62ab4bfd9e3ba4e2b37a2f97cf895c2c WatchSource:0}: Error finding container f1c0a8ca9d2a162d90c9bf2e87eeb23c62ab4bfd9e3ba4e2b37a2f97cf895c2c: Status 404 returned error can't find the container with id f1c0a8ca9d2a162d90c9bf2e87eeb23c62ab4bfd9e3ba4e2b37a2f97cf895c2c Jul 16 13:48:32.658494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:32.658466 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7dljr" Jul 16 13:48:32.717283 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:32.717251 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" event={"ID":"2ed29fd2-7268-429d-acbb-91979daf794f","Type":"ContainerStarted","Data":"f1c0a8ca9d2a162d90c9bf2e87eeb23c62ab4bfd9e3ba4e2b37a2f97cf895c2c"} Jul 16 13:48:33.269438 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:33.269403 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:34.198125 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:34.198092 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:34.301542 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:34.301509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:34.630311 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:34.630273 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:34.797389 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:34.797348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:35.678850 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:35.678808 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:37.748264 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:37.748174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" event={"ID":"2ed29fd2-7268-429d-acbb-91979daf794f","Type":"ContainerStarted","Data":"b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620"} Jul 16 13:48:37.748264 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:37.748264 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:37.770514 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:37.770467 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" podStartSLOduration=1.380041496 podStartE2EDuration="6.770450059s" totalImagesPullingTime="5.390408563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:48:37.768281645 +0000 UTC m=+497.716398377" watchObservedRunningTime="2026-07-16 13:48:37.770450059 +0000 UTC m=+497.718566765" Jul 16 13:48:38.359979 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:38.359941 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:38.461192 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:38.461152 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:40.394210 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.394178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:40.422242 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.422208 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np"] Jul 16 13:48:40.425967 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.425946 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.428523 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.428505 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h2wwc\"" Jul 16 13:48:40.428613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.428513 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 13:48:40.428659 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.428626 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 13:48:40.434934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.434908 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np"] Jul 16 13:48:40.487531 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.487499 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:40.522414 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.522389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.522571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.522452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.522571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.522519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2xb\" (UniqueName: \"kubernetes.io/projected/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-kube-api-access-jg2xb\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.591393 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.591357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:40.623537 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.623502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.623716 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.623562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2xb\" (UniqueName: \"kubernetes.io/projected/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-kube-api-access-jg2xb\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.623716 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.623588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.624347 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.624323 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.626187 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.626162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.632664 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.632640 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2xb\" (UniqueName: \"kubernetes.io/projected/70b52ffa-2f77-4ac6-bfab-9fcb27b3d425-kube-api-access-jg2xb\") pod \"kuadrant-console-plugin-66b5774985-fs8np\" (UID: \"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.706362 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.706278 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gzp4h" Jul 16 13:48:40.736356 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.736324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" Jul 16 13:48:40.875099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:40.875076 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np"] Jul 16 13:48:40.877032 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:48:40.876997 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b52ffa_2f77_4ac6_bfab_9fcb27b3d425.slice/crio-ad37b5cea9f6b43b3fde08a18da4e06021005f16f5169a3d963caa65f6ffac4d WatchSource:0}: Error finding container ad37b5cea9f6b43b3fde08a18da4e06021005f16f5169a3d963caa65f6ffac4d: Status 404 returned error can't find the container with id ad37b5cea9f6b43b3fde08a18da4e06021005f16f5169a3d963caa65f6ffac4d Jul 16 13:48:41.771685 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:41.771653 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" event={"ID":"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425","Type":"ContainerStarted","Data":"ad37b5cea9f6b43b3fde08a18da4e06021005f16f5169a3d963caa65f6ffac4d"} Jul 16 13:48:45.712617 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:45.712581 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:45.782329 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:45.782291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:48.755892 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:48.755831 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:49.911319 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:49.911275 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4"] Jul 16 13:48:49.933571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:49.933540 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4"] Jul 16 13:48:49.933764 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:49.933628 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.013977 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.013928 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.013977 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.013975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xsc\" (UniqueName: \"kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.115311 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.115263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.115311 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.115309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xsc\" (UniqueName: \"kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.116223 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.115695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.124987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.124957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xsc\" (UniqueName: \"kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.247839 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.247752 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:48:50.808838 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.808799 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4"] Jul 16 13:48:50.825683 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.825647 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4"] Jul 16 13:48:50.852840 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.852795 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm"] Jul 16 13:48:50.853119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.853070 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" containerName="manager" containerID="cri-o://b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620" gracePeriod=2 Jul 16 13:48:50.855421 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.855388 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:48:50.867326 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.867304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:50.867812 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.867788 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm"] Jul 16 13:48:50.875491 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:48:50.875365 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2ed29fd2-7268-429d-acbb-91979daf794f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\": pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:50.879164 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.879127 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:48:50.882094 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.882072 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg"] Jul 16 13:48:50.882392 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.882367 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" podUID="53fa5559-14d9-49a1-9986-502214df261f" containerName="manager" containerID="cri-o://0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb" gracePeriod=2 Jul 16 13:48:50.899167 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.899095 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg"] Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.902519 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903124 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" containerName="manager" Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903144 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" containerName="manager" Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903164 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53fa5559-14d9-49a1-9986-502214df261f" containerName="manager" Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903172 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53fa5559-14d9-49a1-9986-502214df261f" containerName="manager" Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903261 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" containerName="manager" Jul 16 13:48:50.904399 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.903273 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53fa5559-14d9-49a1-9986-502214df261f" containerName="manager" Jul 16 13:48:50.908307 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.908250 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:50.912032 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:48:50.911989 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:50.915479 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:48:50.915447 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:48:50.917510 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:48:50.917468 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="53fa5559-14d9-49a1-9986-502214df261f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:48:50.935735 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:50.935709 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:48:51.022786 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.022748 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.022786 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.022786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.023042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.022803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmdg\" (UniqueName: \"kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.023042 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.022821 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscvg\" (UniqueName: \"kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.124341 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.124341 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.124341 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmdg\" (UniqueName: \"kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.124613 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscvg\" (UniqueName: \"kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.124729 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124704 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.124792 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.124743 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.134217 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.134187 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmdg\" (UniqueName: \"kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q62tw\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:51.134375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.134360 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscvg\" (UniqueName: \"kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-55bc7\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.287758 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.287701 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:48:51.292379 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:51.292341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:48:52.846092 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:52.846059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:52.953245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:52.953211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:53.060905 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:53.060851 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:55.024400 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:55.024357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:55.124124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:55.124050 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:55.864506 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:55.864467 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:55.963788 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:55.963751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:57.397105 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:57.397056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:48:57.582058 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:48:57.582021 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:02.186828 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.186801 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.189334 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.189303 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.199272 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.199251 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:02.202570 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.202457 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.205355 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.205311 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="53fa5559-14d9-49a1-9986-502214df261f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:02.206380 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.206361 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:49:02.210434 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:02.210410 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode406e6b7_00bd_4193_b2be_8a203bffbf63.slice/crio-216c37aff72fd782e13803ef5e34b2a19cbb5f2a47d4a979b784dcc0240e4789 WatchSource:0}: Error finding container 216c37aff72fd782e13803ef5e34b2a19cbb5f2a47d4a979b784dcc0240e4789: Status 404 returned error can't find the container with id 216c37aff72fd782e13803ef5e34b2a19cbb5f2a47d4a979b784dcc0240e4789 Jul 16 13:49:02.231185 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.231156 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9dcv\" (UniqueName: \"kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv\") pod \"2ed29fd2-7268-429d-acbb-91979daf794f\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " Jul 16 13:49:02.233858 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.233834 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv" (OuterVolumeSpecName: "kube-api-access-k9dcv") pod "2ed29fd2-7268-429d-acbb-91979daf794f" (UID: "2ed29fd2-7268-429d-acbb-91979daf794f"). InnerVolumeSpecName "kube-api-access-k9dcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:02.234138 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.234115 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:49:02.244097 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:02.244075 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7077ac36_0fce_4138_92de_e55c96c1c7aa.slice/crio-a96a13f7047a8d4bf14bebd7de044c73df9451b1a2465e1d60e8aaff5a048222 WatchSource:0}: Error finding container a96a13f7047a8d4bf14bebd7de044c73df9451b1a2465e1d60e8aaff5a048222: Status 404 returned error can't find the container with id a96a13f7047a8d4bf14bebd7de044c73df9451b1a2465e1d60e8aaff5a048222 Jul 16 13:49:02.332453 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.332419 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn77m\" (UniqueName: \"kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m\") pod \"53fa5559-14d9-49a1-9986-502214df261f\" (UID: \"53fa5559-14d9-49a1-9986-502214df261f\") " Jul 16 13:49:02.332626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.332468 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume\") pod \"2ed29fd2-7268-429d-acbb-91979daf794f\" (UID: \"2ed29fd2-7268-429d-acbb-91979daf794f\") " Jul 16 13:49:02.332626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.332619 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dcv\" (UniqueName: \"kubernetes.io/projected/2ed29fd2-7268-429d-acbb-91979daf794f-kube-api-access-k9dcv\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:02.332988 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.332966 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2ed29fd2-7268-429d-acbb-91979daf794f" (UID: "2ed29fd2-7268-429d-acbb-91979daf794f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:49:02.334452 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.334434 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m" (OuterVolumeSpecName: "kube-api-access-cn77m") pod "53fa5559-14d9-49a1-9986-502214df261f" (UID: "53fa5559-14d9-49a1-9986-502214df261f"). InnerVolumeSpecName "kube-api-access-cn77m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:02.433172 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.433138 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn77m\" (UniqueName: \"kubernetes.io/projected/53fa5559-14d9-49a1-9986-502214df261f-kube-api-access-cn77m\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:02.433172 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.433167 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed29fd2-7268-429d-acbb-91979daf794f-extensions-socket-volume\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:02.474737 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.474704 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" path="/var/lib/kubelet/pods/2ed29fd2-7268-429d-acbb-91979daf794f/volumes" Jul 16 13:49:02.475056 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.475043 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53fa5559-14d9-49a1-9986-502214df261f" path="/var/lib/kubelet/pods/53fa5559-14d9-49a1-9986-502214df261f/volumes" Jul 16 13:49:02.718806 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.718709 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:02.818164 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.818131 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:02.905271 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.905234 2563 generic.go:356] "Generic (PLEG): container finished" podID="2ed29fd2-7268-429d-acbb-91979daf794f" containerID="b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620" exitCode=0 Jul 16 13:49:02.905444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.905299 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.905444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.905343 2563 scope.go:117] "RemoveContainer" containerID="b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620" Jul 16 13:49:02.906798 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.906775 2563 generic.go:356] "Generic (PLEG): container finished" podID="53fa5559-14d9-49a1-9986-502214df261f" containerID="0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb" exitCode=0 Jul 16 13:49:02.906972 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.906828 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:02.907575 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.907506 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.908860 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.908836 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" event={"ID":"e406e6b7-00bd-4193-b2be-8a203bffbf63","Type":"ContainerStarted","Data":"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238"} Jul 16 13:49:02.908993 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.908945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" event={"ID":"e406e6b7-00bd-4193-b2be-8a203bffbf63","Type":"ContainerStarted","Data":"216c37aff72fd782e13803ef5e34b2a19cbb5f2a47d4a979b784dcc0240e4789"} Jul 16 13:49:02.908993 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.908983 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:49:02.909550 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.909523 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.910653 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.910630 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" event={"ID":"70b52ffa-2f77-4ac6-bfab-9fcb27b3d425","Type":"ContainerStarted","Data":"2c3b208a92b71556e587b78e476e4a7b4d01de30c0fe1e4f610956d4e988c75f"} Jul 16 13:49:02.911324 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.911301 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="53fa5559-14d9-49a1-9986-502214df261f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:02.912278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.912259 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" event={"ID":"7077ac36-0fce-4138-92de-e55c96c1c7aa","Type":"ContainerStarted","Data":"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b"} Jul 16 13:49:02.912362 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.912297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" event={"ID":"7077ac36-0fce-4138-92de-e55c96c1c7aa","Type":"ContainerStarted","Data":"a96a13f7047a8d4bf14bebd7de044c73df9451b1a2465e1d60e8aaff5a048222"} Jul 16 13:49:02.912362 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.912318 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:49:02.915490 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.915475 2563 scope.go:117] "RemoveContainer" containerID="b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620" Jul 16 13:49:02.915726 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.915710 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620\": container with ID starting with b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620 not found: ID does not exist" containerID="b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620" Jul 16 13:49:02.915774 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.915735 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620"} err="failed to get container status \"b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620\": rpc error: code = NotFound desc = could not find container \"b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620\": container with ID starting with b157c1046528d62af04bc083a16a66b65b3b70fdcdad8ca03d7121f0545d3620 not found: ID does not exist" Jul 16 13:49:02.915774 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.915751 2563 scope.go:117] "RemoveContainer" containerID="0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb" Jul 16 13:49:02.923662 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.923644 2563 scope.go:117] "RemoveContainer" containerID="0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb" Jul 16 13:49:02.923921 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.923904 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb\": container with ID starting with 0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb not found: ID does not exist" containerID="0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb" Jul 16 13:49:02.923978 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.923927 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb"} err="failed to get container status \"0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb\": rpc error: code = NotFound desc = could not find container \"0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb\": container with ID starting with 0f7771c6e8bb94563a3a60b2ce1593dc1a3c49b75baac7eeac5377abad913feb not found: ID does not exist" Jul 16 13:49:02.932985 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.932936 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" podStartSLOduration=12.932919487 podStartE2EDuration="12.932919487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:02.931362144 +0000 UTC m=+522.879478850" watchObservedRunningTime="2026-07-16 13:49:02.932919487 +0000 UTC m=+522.881036195" Jul 16 13:49:02.933338 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.933314 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.952987 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.952952 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="53fa5559-14d9-49a1-9986-502214df261f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:02.953231 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.953190 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fs8np" podStartSLOduration=1.703913933 podStartE2EDuration="22.953175071s" totalImagesPullingTime="21.249261138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:02.951179787 +0000 UTC m=+522.899296495" watchObservedRunningTime="2026-07-16 13:49:02.953175071 +0000 UTC m=+522.901291777" Jul 16 13:49:02.974794 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.974702 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sbqgm\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="2ed29fd2-7268-429d-acbb-91979daf794f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sbqgm" Jul 16 13:49:02.974974 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:02.974899 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" podStartSLOduration=12.97486977 podStartE2EDuration="12.97486977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:02.97265611 +0000 UTC m=+522.920772839" watchObservedRunningTime="2026-07-16 13:49:02.97486977 +0000 UTC m=+522.922986476" Jul 16 13:49:02.976333 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:02.976311 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vm2kg\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="53fa5559-14d9-49a1-9986-502214df261f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vm2kg" Jul 16 13:49:04.573791 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:04.573749 2563 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-jqhr4_kuadrant-system_76656986-b8e1-492d-9672-5698ab4063f4_0(c2a6877f209bbb5dea6068cd363ed5d5ed1aa1ba26a2be79757a4f0683f215e1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-jqhr4 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 16 13:49:04.574089 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:04.573818 2563 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-jqhr4_kuadrant-system_76656986-b8e1-492d-9672-5698ab4063f4_0(c2a6877f209bbb5dea6068cd363ed5d5ed1aa1ba26a2be79757a4f0683f215e1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-jqhr4 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:04.925301 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.925223 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:04.927564 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:04.927532 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="76656986-b8e1-492d-9672-5698ab4063f4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:04.933775 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.933756 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:04.935989 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:04.935967 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="76656986-b8e1-492d-9672-5698ab4063f4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:04.955478 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.955450 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7xsc\" (UniqueName: \"kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc\") pod \"76656986-b8e1-492d-9672-5698ab4063f4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " Jul 16 13:49:04.955630 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.955541 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume\") pod \"76656986-b8e1-492d-9672-5698ab4063f4\" (UID: \"76656986-b8e1-492d-9672-5698ab4063f4\") " Jul 16 13:49:04.955835 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.955812 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "76656986-b8e1-492d-9672-5698ab4063f4" (UID: "76656986-b8e1-492d-9672-5698ab4063f4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:49:04.957596 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:04.957571 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc" (OuterVolumeSpecName: "kube-api-access-m7xsc") pod "76656986-b8e1-492d-9672-5698ab4063f4" (UID: "76656986-b8e1-492d-9672-5698ab4063f4"). InnerVolumeSpecName "kube-api-access-m7xsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:05.057043 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:05.057014 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xsc\" (UniqueName: \"kubernetes.io/projected/76656986-b8e1-492d-9672-5698ab4063f4-kube-api-access-m7xsc\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:05.057043 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:05.057040 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76656986-b8e1-492d-9672-5698ab4063f4-extensions-socket-volume\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:05.930947 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:05.930916 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:05.932992 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:05.932963 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="76656986-b8e1-492d-9672-5698ab4063f4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:05.938424 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:05.938395 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqhr4\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="76656986-b8e1-492d-9672-5698ab4063f4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqhr4" Jul 16 13:49:06.475028 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:06.474995 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76656986-b8e1-492d-9672-5698ab4063f4" path="/var/lib/kubelet/pods/76656986-b8e1-492d-9672-5698ab4063f4/volumes" Jul 16 13:49:10.547525 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:10.547490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:10.612279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:10.612230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:11.151515 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:11.151478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:11.252239 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:11.252204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:13.921870 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:13.921799 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:49:13.921870 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:13.921864 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:49:13.993642 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:13.993610 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:49:13.993831 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:13.993811 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" podUID="e406e6b7-00bd-4193-b2be-8a203bffbf63" containerName="manager" containerID="cri-o://1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238" gracePeriod=10 Jul 16 13:49:14.246461 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.246435 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 13:49:14.250203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.247829 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:49:14.253620 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.251725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.263224 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.263202 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 13:49:14.338838 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.338810 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhmdg\" (UniqueName: \"kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg\") pod \"e406e6b7-00bd-4193-b2be-8a203bffbf63\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " Jul 16 13:49:14.338990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.338891 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume\") pod \"e406e6b7-00bd-4193-b2be-8a203bffbf63\" (UID: \"e406e6b7-00bd-4193-b2be-8a203bffbf63\") " Jul 16 13:49:14.338990 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.338976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5zn\" (UniqueName: \"kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.339076 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.339059 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.339264 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.339240 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e406e6b7-00bd-4193-b2be-8a203bffbf63" (UID: "e406e6b7-00bd-4193-b2be-8a203bffbf63"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:49:14.340718 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.340703 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg" (OuterVolumeSpecName: "kube-api-access-mhmdg") pod "e406e6b7-00bd-4193-b2be-8a203bffbf63" (UID: "e406e6b7-00bd-4193-b2be-8a203bffbf63"). InnerVolumeSpecName "kube-api-access-mhmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:14.440265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.440228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.440463 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.440292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5zn\" (UniqueName: \"kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.440463 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.440353 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmdg\" (UniqueName: \"kubernetes.io/projected/e406e6b7-00bd-4193-b2be-8a203bffbf63-kube-api-access-mhmdg\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:14.440463 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.440366 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e406e6b7-00bd-4193-b2be-8a203bffbf63-extensions-socket-volume\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:14.440626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.440594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.449750 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.449723 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5zn\" (UniqueName: \"kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn\") pod \"kuadrant-operator-controller-manager-86878c769f-vdb9k\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.564178 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.564147 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.585108 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.585068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:14.699866 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.699823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:14.707951 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.707928 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 13:49:14.709848 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:14.709819 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b8ffa8_839e_4e69_a7d0_fb4d1e6ff585.slice/crio-bcef479d0e9b18cde12d2a0a74272085916066ab29855c3957ef930eacf26c1f WatchSource:0}: Error finding container bcef479d0e9b18cde12d2a0a74272085916066ab29855c3957ef930eacf26c1f: Status 404 returned error can't find the container with id bcef479d0e9b18cde12d2a0a74272085916066ab29855c3957ef930eacf26c1f Jul 16 13:49:14.979579 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.979531 2563 generic.go:356] "Generic (PLEG): container finished" podID="e406e6b7-00bd-4193-b2be-8a203bffbf63" containerID="1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238" exitCode=0 Jul 16 13:49:14.980044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.979597 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" Jul 16 13:49:14.980044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.979620 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" event={"ID":"e406e6b7-00bd-4193-b2be-8a203bffbf63","Type":"ContainerDied","Data":"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238"} Jul 16 13:49:14.980044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.979678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw" event={"ID":"e406e6b7-00bd-4193-b2be-8a203bffbf63","Type":"ContainerDied","Data":"216c37aff72fd782e13803ef5e34b2a19cbb5f2a47d4a979b784dcc0240e4789"} Jul 16 13:49:14.980044 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.979706 2563 scope.go:117] "RemoveContainer" containerID="1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238" Jul 16 13:49:14.981644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.981572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" event={"ID":"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585","Type":"ContainerStarted","Data":"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7"} Jul 16 13:49:14.981644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.981627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" event={"ID":"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585","Type":"ContainerStarted","Data":"bcef479d0e9b18cde12d2a0a74272085916066ab29855c3957ef930eacf26c1f"} Jul 16 13:49:14.981827 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.981672 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:14.988773 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.988758 2563 scope.go:117] "RemoveContainer" containerID="1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238" Jul 16 13:49:14.989073 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:14.989052 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238\": container with ID starting with 1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238 not found: ID does not exist" containerID="1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238" Jul 16 13:49:14.989131 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:14.989083 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238"} err="failed to get container status \"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238\": rpc error: code = NotFound desc = could not find container \"1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238\": container with ID starting with 1064913fa0f2890944cc5d20dcf2daea47ab2e2e74c9c8efe76a81e4c40b2238 not found: ID does not exist" Jul 16 13:49:15.020998 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:15.020973 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:49:15.031581 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:15.031558 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q62tw"] Jul 16 13:49:15.061598 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:15.061547 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" podStartSLOduration=1.061529481 podStartE2EDuration="1.061529481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:15.058169172 +0000 UTC m=+535.006285880" watchObservedRunningTime="2026-07-16 13:49:15.061529481 +0000 UTC m=+535.009646188" Jul 16 13:49:16.474834 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:16.474786 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e406e6b7-00bd-4193-b2be-8a203bffbf63" path="/var/lib/kubelet/pods/e406e6b7-00bd-4193-b2be-8a203bffbf63/volumes" Jul 16 13:49:16.959291 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:16.959256 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:17.062177 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:17.062147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:21.639260 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:21.639222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:21.799065 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:21.799030 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:25.991266 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:25.991234 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 13:49:26.040943 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.040911 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:49:26.041200 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.041173 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" podUID="7077ac36-0fce-4138-92de-e55c96c1c7aa" containerName="manager" containerID="cri-o://377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b" gracePeriod=10 Jul 16 13:49:26.808222 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.808198 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:49:26.947933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.947836 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kscvg\" (UniqueName: \"kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg\") pod \"7077ac36-0fce-4138-92de-e55c96c1c7aa\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " Jul 16 13:49:26.947933 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.947892 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume\") pod \"7077ac36-0fce-4138-92de-e55c96c1c7aa\" (UID: \"7077ac36-0fce-4138-92de-e55c96c1c7aa\") " Jul 16 13:49:26.948290 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.948265 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7077ac36-0fce-4138-92de-e55c96c1c7aa" (UID: "7077ac36-0fce-4138-92de-e55c96c1c7aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:49:26.949970 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:26.949943 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg" (OuterVolumeSpecName: "kube-api-access-kscvg") pod "7077ac36-0fce-4138-92de-e55c96c1c7aa" (UID: "7077ac36-0fce-4138-92de-e55c96c1c7aa"). InnerVolumeSpecName "kube-api-access-kscvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:27.046606 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.046565 2563 generic.go:356] "Generic (PLEG): container finished" podID="7077ac36-0fce-4138-92de-e55c96c1c7aa" containerID="377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b" exitCode=0 Jul 16 13:49:27.046991 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.046609 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" Jul 16 13:49:27.046991 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.046644 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" event={"ID":"7077ac36-0fce-4138-92de-e55c96c1c7aa","Type":"ContainerDied","Data":"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b"} Jul 16 13:49:27.046991 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.046703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7" event={"ID":"7077ac36-0fce-4138-92de-e55c96c1c7aa","Type":"ContainerDied","Data":"a96a13f7047a8d4bf14bebd7de044c73df9451b1a2465e1d60e8aaff5a048222"} Jul 16 13:49:27.046991 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.046733 2563 scope.go:117] "RemoveContainer" containerID="377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b" Jul 16 13:49:27.048467 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.048449 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscvg\" (UniqueName: \"kubernetes.io/projected/7077ac36-0fce-4138-92de-e55c96c1c7aa-kube-api-access-kscvg\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:27.048551 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.048472 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7077ac36-0fce-4138-92de-e55c96c1c7aa-extensions-socket-volume\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:27.055966 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.055950 2563 scope.go:117] "RemoveContainer" containerID="377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b" Jul 16 13:49:27.056211 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:27.056189 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b\": container with ID starting with 377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b not found: ID does not exist" containerID="377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b" Jul 16 13:49:27.056276 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.056220 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b"} err="failed to get container status \"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b\": rpc error: code = NotFound desc = could not find container \"377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b\": container with ID starting with 377a9d8b1b01e555134f10976ab59f995c5046553722b92b5c7584bbffe8b11b not found: ID does not exist" Jul 16 13:49:27.068361 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.068336 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:49:27.075071 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:27.075050 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-55bc7"] Jul 16 13:49:28.473920 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:28.473837 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7077ac36-0fce-4138-92de-e55c96c1c7aa" path="/var/lib/kubelet/pods/7077ac36-0fce-4138-92de-e55c96c1c7aa/volumes" Jul 16 13:49:30.558636 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.558604 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247"] Jul 16 13:49:30.559095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559012 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7077ac36-0fce-4138-92de-e55c96c1c7aa" containerName="manager" Jul 16 13:49:30.559095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559025 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7077ac36-0fce-4138-92de-e55c96c1c7aa" containerName="manager" Jul 16 13:49:30.559095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559058 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e406e6b7-00bd-4193-b2be-8a203bffbf63" containerName="manager" Jul 16 13:49:30.559095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559064 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e406e6b7-00bd-4193-b2be-8a203bffbf63" containerName="manager" Jul 16 13:49:30.559226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559126 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e406e6b7-00bd-4193-b2be-8a203bffbf63" containerName="manager" Jul 16 13:49:30.559226 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.559137 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7077ac36-0fce-4138-92de-e55c96c1c7aa" containerName="manager" Jul 16 13:49:30.583508 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.583473 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:30.610085 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.610048 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247"] Jul 16 13:49:30.610270 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.610178 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.612409 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.612388 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 13:49:30.612516 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.612390 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kcnj8\"" Jul 16 13:49:30.682566 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682566 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682571 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682823 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0054dc-68ba-4482-a4b6-138f93c75a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.682913 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.682898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbx2\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-kube-api-access-8lbx2\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.783821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.783782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.783821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.783823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.783956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.783992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0054dc-68ba-4482-a4b6-138f93c75a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784063 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbx2\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-kube-api-access-8lbx2\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784244 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784244 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784244 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784250 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784387 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784290 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784481 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784542 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784515 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.784897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.784842 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0054dc-68ba-4482-a4b6-138f93c75a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.786381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.786354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.786516 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.786500 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.791906 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.791867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.792118 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.792100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbx2\" (UniqueName: \"kubernetes.io/projected/0a0054dc-68ba-4482-a4b6-138f93c75a51-kube-api-access-8lbx2\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6247\" (UID: \"0a0054dc-68ba-4482-a4b6-138f93c75a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:30.883678 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.883576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:30.921095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:30.921065 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:31.056973 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:31.056946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247"] Jul 16 13:49:31.058758 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:31.058734 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0054dc_68ba_4482_a4b6_138f93c75a51.slice/crio-656038cf2cbc3a8e20c92e6fbf7d8646437814353ab2d1df5c44c73e85b7aac1 WatchSource:0}: Error finding container 656038cf2cbc3a8e20c92e6fbf7d8646437814353ab2d1df5c44c73e85b7aac1: Status 404 returned error can't find the container with id 656038cf2cbc3a8e20c92e6fbf7d8646437814353ab2d1df5c44c73e85b7aac1 Jul 16 13:49:31.069971 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:31.069947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" event={"ID":"0a0054dc-68ba-4482-a4b6-138f93c75a51","Type":"ContainerStarted","Data":"656038cf2cbc3a8e20c92e6fbf7d8646437814353ab2d1df5c44c73e85b7aac1"} Jul 16 13:49:33.839674 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:33.839639 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:49:33.840003 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:33.839710 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:49:33.840003 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:33.839742 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:49:34.093051 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:34.092967 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" event={"ID":"0a0054dc-68ba-4482-a4b6-138f93c75a51","Type":"ContainerStarted","Data":"aee1b0e2e1ddea4cfb5b4444f055c6de4d552ee4b618f6e53a0276a065606e9f"} Jul 16 13:49:34.113948 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:34.113889 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podStartSLOduration=1.335313198 podStartE2EDuration="4.113858097s" totalImagesPullingTime="2.778544899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:34.112323465 +0000 UTC m=+554.060440233" watchObservedRunningTime="2026-07-16 13:49:34.113858097 +0000 UTC m=+554.061974803" Jul 16 13:49:34.921662 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:34.921625 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:34.926659 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:34.926638 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:35.098254 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:35.098218 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:35.099125 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:35.099106 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:49:42.989726 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:42.989685 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:49:42.992357 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:42.992314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:43.008342 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.008309 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:49:43.008490 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.008425 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.010824 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.010802 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 13:49:43.083829 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.083801 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:49:43.092841 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.092807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:43.102903 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.102857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.103068 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.102983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztx7t\" (UniqueName: \"kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.162434 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.162395 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:43.203603 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.203573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx7t\" (UniqueName: \"kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.203807 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.203688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.204444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.204418 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.212055 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.212034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx7t\" (UniqueName: \"kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t\") pod \"limitador-limitador-64c94fd64f-jhfff\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.318890 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.318848 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:43.465195 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.465169 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:49:43.467366 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:43.467340 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ee10c3_7fc5_4288_a621_c6a7b921c205.slice/crio-f92b66afd1c51f3b60809d5199be786e53ff51583666905a4217fa0a8fb985f5 WatchSource:0}: Error finding container f92b66afd1c51f3b60809d5199be786e53ff51583666905a4217fa0a8fb985f5: Status 404 returned error can't find the container with id f92b66afd1c51f3b60809d5199be786e53ff51583666905a4217fa0a8fb985f5 Jul 16 13:49:43.482861 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.482834 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:49:43.487697 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.487675 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.495283 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.495256 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:49:43.525622 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.525587 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:49:43.607201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.607115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr2v\" (UniqueName: \"kubernetes.io/projected/9511c1dd-cfb2-494d-859b-908fa156e867-kube-api-access-rqr2v\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.607392 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.607373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9511c1dd-cfb2-494d-859b-908fa156e867-config-file\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.708989 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.708950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9511c1dd-cfb2-494d-859b-908fa156e867-config-file\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.709171 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.709065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr2v\" (UniqueName: \"kubernetes.io/projected/9511c1dd-cfb2-494d-859b-908fa156e867-kube-api-access-rqr2v\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.709622 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.709601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9511c1dd-cfb2-494d-859b-908fa156e867-config-file\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.718255 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.718229 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr2v\" (UniqueName: \"kubernetes.io/projected/9511c1dd-cfb2-494d-859b-908fa156e867-kube-api-access-rqr2v\") pod \"limitador-limitador-54d6d988d5-7w8bs\" (UID: \"9511c1dd-cfb2-494d-859b-908fa156e867\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.799501 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.799200 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:43.801754 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.801730 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:43.806098 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.806076 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:43.808596 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.808575 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kvc2h\"" Jul 16 13:49:43.816900 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.816856 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:43.837777 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.837731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:43.909577 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.909542 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:49:43.911626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.911600 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mvc\" (UniqueName: \"kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc\") pod \"authorino-58cd955675-rz896\" (UID: \"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d\") " pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:43.950424 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.950400 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:49:43.952370 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:43.952345 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9511c1dd_cfb2_494d_859b_908fa156e867.slice/crio-14d1bd8c2679cccda443147d7663833c5949fbbf541afb7e3cbd1a88944f32b5 WatchSource:0}: Error finding container 14d1bd8c2679cccda443147d7663833c5949fbbf541afb7e3cbd1a88944f32b5: Status 404 returned error can't find the container with id 14d1bd8c2679cccda443147d7663833c5949fbbf541afb7e3cbd1a88944f32b5 Jul 16 13:49:43.997865 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:43.997830 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:49:44.000361 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.000332 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:49:44.009188 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.009163 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:49:44.012682 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.012656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mvc\" (UniqueName: \"kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc\") pod \"authorino-58cd955675-rz896\" (UID: \"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d\") " pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:44.021867 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.021840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mvc\" (UniqueName: \"kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc\") pod \"authorino-58cd955675-rz896\" (UID: \"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d\") " pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:44.113712 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.113615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7lv\" (UniqueName: \"kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv\") pod \"authorino-75d556bd77-xjtwq\" (UID: \"feb27861-4754-4c18-a581-938ac49da7ec\") " pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:49:44.118014 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.117989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:44.145467 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.145430 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" event={"ID":"f0ee10c3-7fc5-4288-a621-c6a7b921c205","Type":"ContainerStarted","Data":"f92b66afd1c51f3b60809d5199be786e53ff51583666905a4217fa0a8fb985f5"} Jul 16 13:49:44.146825 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.146797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" event={"ID":"9511c1dd-cfb2-494d-859b-908fa156e867","Type":"ContainerStarted","Data":"14d1bd8c2679cccda443147d7663833c5949fbbf541afb7e3cbd1a88944f32b5"} Jul 16 13:49:44.218532 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.217231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7lv\" (UniqueName: \"kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv\") pod \"authorino-75d556bd77-xjtwq\" (UID: \"feb27861-4754-4c18-a581-938ac49da7ec\") " pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:49:44.229099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.228757 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7lv\" (UniqueName: \"kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv\") pod \"authorino-75d556bd77-xjtwq\" (UID: \"feb27861-4754-4c18-a581-938ac49da7ec\") " pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:49:44.280190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.280104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:44.283750 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:44.283404 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04ebe9f8_6f5f_49a8_8c75_c70ad0d98c7d.slice/crio-afa5f8182e6018b9d9942a4a914029ce2be45341d44bb80114e557f56458b1fe WatchSource:0}: Error finding container afa5f8182e6018b9d9942a4a914029ce2be45341d44bb80114e557f56458b1fe: Status 404 returned error can't find the container with id afa5f8182e6018b9d9942a4a914029ce2be45341d44bb80114e557f56458b1fe Jul 16 13:49:44.310568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.310538 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:49:44.507694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:44.507657 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:49:44.514473 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:49:44.514440 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb27861_4754_4c18_a581_938ac49da7ec.slice/crio-318f412e65c542a8b5d50a8ae7143e4eb26c32e182a5ee4abba6fc9ca78cf049 WatchSource:0}: Error finding container 318f412e65c542a8b5d50a8ae7143e4eb26c32e182a5ee4abba6fc9ca78cf049: Status 404 returned error can't find the container with id 318f412e65c542a8b5d50a8ae7143e4eb26c32e182a5ee4abba6fc9ca78cf049 Jul 16 13:49:45.185708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:45.185650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rz896" event={"ID":"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d","Type":"ContainerStarted","Data":"afa5f8182e6018b9d9942a4a914029ce2be45341d44bb80114e557f56458b1fe"} Jul 16 13:49:45.189600 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:45.189532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xjtwq" event={"ID":"feb27861-4754-4c18-a581-938ac49da7ec","Type":"ContainerStarted","Data":"318f412e65c542a8b5d50a8ae7143e4eb26c32e182a5ee4abba6fc9ca78cf049"} Jul 16 13:49:49.231334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.231284 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xjtwq" event={"ID":"feb27861-4754-4c18-a581-938ac49da7ec","Type":"ContainerStarted","Data":"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586"} Jul 16 13:49:49.233084 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.233036 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" event={"ID":"f0ee10c3-7fc5-4288-a621-c6a7b921c205","Type":"ContainerStarted","Data":"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894"} Jul 16 13:49:49.233227 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.233126 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:49:49.234728 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.234694 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" event={"ID":"9511c1dd-cfb2-494d-859b-908fa156e867","Type":"ContainerStarted","Data":"e3a6b935dfb98c4d129f0224e6bdf6e449cb4c9f69027a4b8405f8cb5cc36b60"} Jul 16 13:49:49.234900 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.234781 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:49:49.236217 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.236153 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rz896" event={"ID":"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d","Type":"ContainerStarted","Data":"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e"} Jul 16 13:49:49.249457 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.249414 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-xjtwq" podStartSLOduration=1.760870654 podStartE2EDuration="6.249400125s" totalImagesPullingTime="4.488529471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:49.247227334 +0000 UTC m=+569.195344052" watchObservedRunningTime="2026-07-16 13:49:49.249400125 +0000 UTC m=+569.197516831" Jul 16 13:49:49.273131 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.272862 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" podStartSLOduration=1.735210395 podStartE2EDuration="7.27284574s" totalImagesPullingTime="5.537635345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:49.270924919 +0000 UTC m=+569.219041647" watchObservedRunningTime="2026-07-16 13:49:49.27284574 +0000 UTC m=+569.220962448" Jul 16 13:49:49.280479 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.280456 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:49.297977 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.297932 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rz896" podStartSLOduration=1.579445277 podStartE2EDuration="6.297915898s" totalImagesPullingTime="4.718470621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:49.295914201 +0000 UTC m=+569.244030908" watchObservedRunningTime="2026-07-16 13:49:49.297915898 +0000 UTC m=+569.246032604" Jul 16 13:49:49.319426 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:49.319375 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" podStartSLOduration=1.249677358 podStartE2EDuration="6.319359989s" totalImagesPullingTime="5.069682631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:49:49.317492403 +0000 UTC m=+569.265609111" watchObservedRunningTime="2026-07-16 13:49:49.319359989 +0000 UTC m=+569.267476696" Jul 16 13:49:51.247272 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:51.247228 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rz896" podUID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" containerName="authorino" containerID="cri-o://8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e" gracePeriod=30 Jul 16 13:49:51.505748 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:51.505672 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:51.595518 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:51.595476 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5mvc\" (UniqueName: \"kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc\") pod \"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d\" (UID: \"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d\") " Jul 16 13:49:51.597594 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:51.597561 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc" (OuterVolumeSpecName: "kube-api-access-f5mvc") pod "04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" (UID: "04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d"). InnerVolumeSpecName "kube-api-access-f5mvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:49:51.696836 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:51.696801 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5mvc\" (UniqueName: \"kubernetes.io/projected/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d-kube-api-access-f5mvc\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:49:52.253828 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.253792 2563 generic.go:356] "Generic (PLEG): container finished" podID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" containerID="8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e" exitCode=0 Jul 16 13:49:52.254230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.253840 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rz896" Jul 16 13:49:52.254230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.253897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rz896" event={"ID":"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d","Type":"ContainerDied","Data":"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e"} Jul 16 13:49:52.254230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.253944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rz896" event={"ID":"04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d","Type":"ContainerDied","Data":"afa5f8182e6018b9d9942a4a914029ce2be45341d44bb80114e557f56458b1fe"} Jul 16 13:49:52.254230 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.253961 2563 scope.go:117] "RemoveContainer" containerID="8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e" Jul 16 13:49:52.262805 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.262786 2563 scope.go:117] "RemoveContainer" containerID="8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e" Jul 16 13:49:52.263105 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:49:52.263085 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e\": container with ID starting with 8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e not found: ID does not exist" containerID="8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e" Jul 16 13:49:52.263193 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.263113 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e"} err="failed to get container status \"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e\": rpc error: code = NotFound desc = could not find container \"8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e\": container with ID starting with 8a576e0b66d437f42a3bffb99eb2d862dc6d8a549bd9cd41e3bf069285ba5c2e not found: ID does not exist" Jul 16 13:49:52.275924 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.275889 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:52.279554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.279527 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rz896"] Jul 16 13:49:52.475029 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:49:52.474996 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" path="/var/lib/kubelet/pods/04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d/volumes" Jul 16 13:50:00.245073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.245043 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:50:00.245495 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.245110 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7w8bs" Jul 16 13:50:00.308784 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.308752 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:50:00.309021 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.308993 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" podUID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" containerName="limitador" containerID="cri-o://b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894" gracePeriod=30 Jul 16 13:50:00.873587 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.873515 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:50:00.979626 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.979590 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztx7t\" (UniqueName: \"kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t\") pod \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " Jul 16 13:50:00.979805 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.979649 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file\") pod \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\" (UID: \"f0ee10c3-7fc5-4288-a621-c6a7b921c205\") " Jul 16 13:50:00.980087 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.980062 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file" (OuterVolumeSpecName: "config-file") pod "f0ee10c3-7fc5-4288-a621-c6a7b921c205" (UID: "f0ee10c3-7fc5-4288-a621-c6a7b921c205"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:50:00.981682 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:00.981664 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t" (OuterVolumeSpecName: "kube-api-access-ztx7t") pod "f0ee10c3-7fc5-4288-a621-c6a7b921c205" (UID: "f0ee10c3-7fc5-4288-a621-c6a7b921c205"). InnerVolumeSpecName "kube-api-access-ztx7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:50:01.081156 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.081119 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztx7t\" (UniqueName: \"kubernetes.io/projected/f0ee10c3-7fc5-4288-a621-c6a7b921c205-kube-api-access-ztx7t\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:01.081156 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.081150 2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f0ee10c3-7fc5-4288-a621-c6a7b921c205-config-file\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:01.306265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.306232 2563 generic.go:356] "Generic (PLEG): container finished" podID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" containerID="b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894" exitCode=0 Jul 16 13:50:01.306652 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.306276 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" Jul 16 13:50:01.306652 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.306318 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" event={"ID":"f0ee10c3-7fc5-4288-a621-c6a7b921c205","Type":"ContainerDied","Data":"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894"} Jul 16 13:50:01.306652 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.306369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jhfff" event={"ID":"f0ee10c3-7fc5-4288-a621-c6a7b921c205","Type":"ContainerDied","Data":"f92b66afd1c51f3b60809d5199be786e53ff51583666905a4217fa0a8fb985f5"} Jul 16 13:50:01.306652 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.306392 2563 scope.go:117] "RemoveContainer" containerID="b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894" Jul 16 13:50:01.316143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.315926 2563 scope.go:117] "RemoveContainer" containerID="b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894" Jul 16 13:50:01.316207 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:50:01.316176 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894\": container with ID starting with b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894 not found: ID does not exist" containerID="b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894" Jul 16 13:50:01.316247 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.316202 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894"} err="failed to get container status \"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894\": rpc error: code = NotFound desc = could not find container \"b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894\": container with ID starting with b4e8cda00044c0bdaa269d792691e5fb634e2babb3b1da90c102d59c3ba10894 not found: ID does not exist" Jul 16 13:50:01.328785 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.328755 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:50:01.332731 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.332709 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jhfff"] Jul 16 13:50:01.894493 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.894447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:01.985639 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:01.985603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:02.475346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.475310 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" path="/var/lib/kubelet/pods/f0ee10c3-7fc5-4288-a621-c6a7b921c205/volumes" Jul 16 13:50:02.495955 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.495922 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d48v7"] Jul 16 13:50:02.496427 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496412 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" containerName="authorino" Jul 16 13:50:02.496477 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496433 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" containerName="authorino" Jul 16 13:50:02.496530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496481 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" containerName="limitador" Jul 16 13:50:02.496530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496488 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" containerName="limitador" Jul 16 13:50:02.496631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496555 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ee10c3-7fc5-4288-a621-c6a7b921c205" containerName="limitador" Jul 16 13:50:02.496631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.496568 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ebe9f8-6f5f-49a8-8c75-c70ad0d98c7d" containerName="authorino" Jul 16 13:50:02.499157 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.499138 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.501914 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.501889 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 13:50:02.502040 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.501927 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 13:50:02.502260 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.502246 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 16 13:50:02.505796 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.505778 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4td9m\"" Jul 16 13:50:02.517346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.517319 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d48v7"] Jul 16 13:50:02.547444 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.547415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:02.595916 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.595866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cfeb2d-2092-451e-a468-c68dd62f3d18-data\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.596110 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.595985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2z8\" (UniqueName: \"kubernetes.io/projected/c0cfeb2d-2092-451e-a468-c68dd62f3d18-kube-api-access-qk2z8\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.696989 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.696954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cfeb2d-2092-451e-a468-c68dd62f3d18-data\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.697151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.697004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2z8\" (UniqueName: \"kubernetes.io/projected/c0cfeb2d-2092-451e-a468-c68dd62f3d18-kube-api-access-qk2z8\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.697448 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.697430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cfeb2d-2092-451e-a468-c68dd62f3d18-data\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.721553 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.721525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2z8\" (UniqueName: \"kubernetes.io/projected/c0cfeb2d-2092-451e-a468-c68dd62f3d18-kube-api-access-qk2z8\") pod \"postgres-69fb4594cf-d48v7\" (UID: \"c0cfeb2d-2092-451e-a468-c68dd62f3d18\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.808793 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.808767 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:02.950332 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:02.950288 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d48v7"] Jul 16 13:50:02.953107 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:50:02.953082 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cfeb2d_2092_451e_a468_c68dd62f3d18.slice/crio-2d02eb8575f7ad2068f7afa9476e5f5e525e60242ca7611f151c07e0cff42998 WatchSource:0}: Error finding container 2d02eb8575f7ad2068f7afa9476e5f5e525e60242ca7611f151c07e0cff42998: Status 404 returned error can't find the container with id 2d02eb8575f7ad2068f7afa9476e5f5e525e60242ca7611f151c07e0cff42998 Jul 16 13:50:03.319299 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:03.319263 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" event={"ID":"c0cfeb2d-2092-451e-a468-c68dd62f3d18","Type":"ContainerStarted","Data":"2d02eb8575f7ad2068f7afa9476e5f5e525e60242ca7611f151c07e0cff42998"} Jul 16 13:50:04.308052 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:04.308019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:04.371354 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:04.371306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:09.359760 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:09.359724 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" event={"ID":"c0cfeb2d-2092-451e-a468-c68dd62f3d18","Type":"ContainerStarted","Data":"4b9de4b9f999c6fb23876a968ff4a448f0f917406bf9c5368f557d509994d24f"} Jul 16 13:50:09.360153 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:09.359837 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:09.380708 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:09.379859 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" podStartSLOduration=1.555977634 podStartE2EDuration="7.379833371s" totalImagesPullingTime="5.823855737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:50:09.375729264 +0000 UTC m=+589.323845985" watchObservedRunningTime="2026-07-16 13:50:09.379833371 +0000 UTC m=+589.327950079" Jul 16 13:50:15.396585 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:15.396559 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d48v7" Jul 16 13:50:15.737649 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:15.737549 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:16.017154 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.017119 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lwfm7"] Jul 16 13:50:16.020771 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.020750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.032929 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.032898 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lwfm7"] Jul 16 13:50:16.079481 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.079451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:16.125489 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.125453 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkcq\" (UniqueName: \"kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq\") pod \"authorino-5d4b67c446-lwfm7\" (UID: \"dcac34f0-6322-4481-b267-6b4f89e412fe\") " pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.189435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.189401 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lwfm7"] Jul 16 13:50:16.189688 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:50:16.189668 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tkkcq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-lwfm7" podUID="dcac34f0-6322-4481-b267-6b4f89e412fe" Jul 16 13:50:16.226331 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.226294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkcq\" (UniqueName: \"kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq\") pod \"authorino-5d4b67c446-lwfm7\" (UID: \"dcac34f0-6322-4481-b267-6b4f89e412fe\") " pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.249564 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.249530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkcq\" (UniqueName: \"kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq\") pod \"authorino-5d4b67c446-lwfm7\" (UID: \"dcac34f0-6322-4481-b267-6b4f89e412fe\") " pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.269780 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.269702 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-mnmpc"] Jul 16 13:50:16.274955 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.274613 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.278190 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.278169 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 13:50:16.282498 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.282472 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-mnmpc"] Jul 16 13:50:16.327391 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.327354 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-mnmpc"] Jul 16 13:50:16.327644 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:50:16.327612 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7whfd tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-7whfd tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-mnmpc" podUID="178c0c7c-c319-444f-b62b-18ba5d73dbd6" Jul 16 13:50:16.365715 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.365674 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:50:16.369176 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.369158 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.377745 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.377720 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:50:16.399107 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.399083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.399107 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.399096 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.407113 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.407090 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.410345 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.410325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:16.427979 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.427949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.428154 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.428071 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whfd\" (UniqueName: \"kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.529161 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.529077 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkkcq\" (UniqueName: \"kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq\") pod \"dcac34f0-6322-4481-b267-6b4f89e412fe\" (UID: \"dcac34f0-6322-4481-b267-6b4f89e412fe\") " Jul 16 13:50:16.529346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.529228 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.529346 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.529283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whfd\" (UniqueName: \"kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.529476 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.529358 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89pm\" (UniqueName: \"kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.529476 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.529430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.531265 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.531236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq" (OuterVolumeSpecName: "kube-api-access-tkkcq") pod "dcac34f0-6322-4481-b267-6b4f89e412fe" (UID: "dcac34f0-6322-4481-b267-6b4f89e412fe"). InnerVolumeSpecName "kube-api-access-tkkcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:50:16.531855 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.531831 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.537220 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.537200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whfd\" (UniqueName: \"kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd\") pod \"authorino-76877994-mnmpc\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:16.629819 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.629782 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7whfd\" (UniqueName: \"kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd\") pod \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " Jul 16 13:50:16.630018 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.629839 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert\") pod \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\" (UID: \"178c0c7c-c319-444f-b62b-18ba5d73dbd6\") " Jul 16 13:50:16.630161 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.630139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89pm\" (UniqueName: \"kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.630250 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.630230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.630381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.630274 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkkcq\" (UniqueName: \"kubernetes.io/projected/dcac34f0-6322-4481-b267-6b4f89e412fe-kube-api-access-tkkcq\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:16.632007 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.631983 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "178c0c7c-c319-444f-b62b-18ba5d73dbd6" (UID: "178c0c7c-c319-444f-b62b-18ba5d73dbd6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:50:16.632115 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.632017 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd" (OuterVolumeSpecName: "kube-api-access-7whfd") pod "178c0c7c-c319-444f-b62b-18ba5d73dbd6" (UID: "178c0c7c-c319-444f-b62b-18ba5d73dbd6"). InnerVolumeSpecName "kube-api-access-7whfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:50:16.632439 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.632423 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.638520 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.638497 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89pm\" (UniqueName: \"kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm\") pod \"authorino-d79d76977-ppxmt\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.679530 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.679495 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:50:16.731309 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.731148 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7whfd\" (UniqueName: \"kubernetes.io/projected/178c0c7c-c319-444f-b62b-18ba5d73dbd6-kube-api-access-7whfd\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:16.731309 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.731179 2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178c0c7c-c319-444f-b62b-18ba5d73dbd6-tls-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:16.814133 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:16.814105 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:50:16.815887 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:50:16.815843 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0066bed2_dcf0_4de3_a7f4_4feafaf90c60.slice/crio-e7fb376919b1612c0dd861c04747785fa097acdda6f4b6420148e56a2fa03403 WatchSource:0}: Error finding container e7fb376919b1612c0dd861c04747785fa097acdda6f4b6420148e56a2fa03403: Status 404 returned error can't find the container with id e7fb376919b1612c0dd861c04747785fa097acdda6f4b6420148e56a2fa03403 Jul 16 13:50:17.405924 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.405866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d79d76977-ppxmt" event={"ID":"0066bed2-dcf0-4de3-a7f4-4feafaf90c60","Type":"ContainerStarted","Data":"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90"} Jul 16 13:50:17.406375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.405938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d79d76977-ppxmt" event={"ID":"0066bed2-dcf0-4de3-a7f4-4feafaf90c60","Type":"ContainerStarted","Data":"e7fb376919b1612c0dd861c04747785fa097acdda6f4b6420148e56a2fa03403"} Jul 16 13:50:17.406375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.405950 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lwfm7" Jul 16 13:50:17.406375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.406066 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mnmpc" Jul 16 13:50:17.422445 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.422396 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d79d76977-ppxmt" podStartSLOduration=1.036061994 podStartE2EDuration="1.422377945s" totalImagesPullingTime="386.315951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:50:17.420523782 +0000 UTC m=+597.368640481" watchObservedRunningTime="2026-07-16 13:50:17.422377945 +0000 UTC m=+597.370494652" Jul 16 13:50:17.446111 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.446067 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lwfm7"] Jul 16 13:50:17.448844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.448819 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lwfm7"] Jul 16 13:50:17.451003 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.450980 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:50:17.451192 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.451170 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-xjtwq" podUID="feb27861-4754-4c18-a581-938ac49da7ec" containerName="authorino" containerID="cri-o://d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586" gracePeriod=30 Jul 16 13:50:17.474245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.474202 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-mnmpc"] Jul 16 13:50:17.478097 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.478072 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-mnmpc"] Jul 16 13:50:17.700854 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.700828 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:50:17.843203 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.843170 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w7lv\" (UniqueName: \"kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv\") pod \"feb27861-4754-4c18-a581-938ac49da7ec\" (UID: \"feb27861-4754-4c18-a581-938ac49da7ec\") " Jul 16 13:50:17.845266 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.845237 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv" (OuterVolumeSpecName: "kube-api-access-9w7lv") pod "feb27861-4754-4c18-a581-938ac49da7ec" (UID: "feb27861-4754-4c18-a581-938ac49da7ec"). InnerVolumeSpecName "kube-api-access-9w7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:50:17.944101 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:17.944065 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7lv\" (UniqueName: \"kubernetes.io/projected/feb27861-4754-4c18-a581-938ac49da7ec-kube-api-access-9w7lv\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:50:18.418099 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.418060 2563 generic.go:356] "Generic (PLEG): container finished" podID="feb27861-4754-4c18-a581-938ac49da7ec" containerID="d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586" exitCode=0 Jul 16 13:50:18.418568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.418110 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xjtwq" Jul 16 13:50:18.418568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.418142 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xjtwq" event={"ID":"feb27861-4754-4c18-a581-938ac49da7ec","Type":"ContainerDied","Data":"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586"} Jul 16 13:50:18.418568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.418192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xjtwq" event={"ID":"feb27861-4754-4c18-a581-938ac49da7ec","Type":"ContainerDied","Data":"318f412e65c542a8b5d50a8ae7143e4eb26c32e182a5ee4abba6fc9ca78cf049"} Jul 16 13:50:18.418568 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.418216 2563 scope.go:117] "RemoveContainer" containerID="d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586" Jul 16 13:50:18.427809 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.427792 2563 scope.go:117] "RemoveContainer" containerID="d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586" Jul 16 13:50:18.428080 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:50:18.428060 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586\": container with ID starting with d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586 not found: ID does not exist" containerID="d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586" Jul 16 13:50:18.428154 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.428088 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586"} err="failed to get container status \"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586\": rpc error: code = NotFound desc = could not find container \"d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586\": container with ID starting with d60c6410e17cbce0e5b75b8b07b6cd023b5e36d1ee9d08bdc1123984f684a586 not found: ID does not exist" Jul 16 13:50:18.439836 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.439814 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:50:18.447464 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.447443 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xjtwq"] Jul 16 13:50:18.474781 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.474752 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178c0c7c-c319-444f-b62b-18ba5d73dbd6" path="/var/lib/kubelet/pods/178c0c7c-c319-444f-b62b-18ba5d73dbd6/volumes" Jul 16 13:50:18.475025 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.475006 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcac34f0-6322-4481-b267-6b4f89e412fe" path="/var/lib/kubelet/pods/dcac34f0-6322-4481-b267-6b4f89e412fe/volumes" Jul 16 13:50:18.475245 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:18.475232 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb27861-4754-4c18-a581-938ac49da7ec" path="/var/lib/kubelet/pods/feb27861-4754-4c18-a581-938ac49da7ec/volumes" Jul 16 13:50:20.634087 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:20.633982 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:50:20.657306 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:20.634480 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:50:23.613076 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:23.613029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:23.714277 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:23.714241 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:27.447789 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:27.447744 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:27.545614 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:27.545583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:58.265350 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:58.265311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:58.478327 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:58.478298 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:58.768037 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:58.767997 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:58.934962 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:58.934933 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:59.029293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.029200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:59.425952 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.425918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:59.590440 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.590355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:50:59.641810 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.641784 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-wswqx"] Jul 16 13:50:59.642256 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.642242 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb27861-4754-4c18-a581-938ac49da7ec" containerName="authorino" Jul 16 13:50:59.642324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.642258 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb27861-4754-4c18-a581-938ac49da7ec" containerName="authorino" Jul 16 13:50:59.642324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.642319 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb27861-4754-4c18-a581-938ac49da7ec" containerName="authorino" Jul 16 13:50:59.645494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.645433 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.647542 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.647520 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 13:50:59.658161 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.658137 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-wswqx"] Jul 16 13:50:59.726926 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.726863 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6224b78-f7ed-435e-9283-5fcf886e4e5d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.727117 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.726952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p2g\" (UniqueName: \"kubernetes.io/projected/b6224b78-f7ed-435e-9283-5fcf886e4e5d-kube-api-access-t9p2g\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.817495 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.817464 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:50:59.828262 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.828232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6224b78-f7ed-435e-9283-5fcf886e4e5d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.828417 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.828279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p2g\" (UniqueName: \"kubernetes.io/projected/b6224b78-f7ed-435e-9283-5fcf886e4e5d-kube-api-access-t9p2g\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.828940 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.828919 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6224b78-f7ed-435e-9283-5fcf886e4e5d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.837279 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.837256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p2g\" (UniqueName: \"kubernetes.io/projected/b6224b78-f7ed-435e-9283-5fcf886e4e5d-kube-api-access-t9p2g\") pod \"payload-processing-f9b8d9b9b-wswqx\" (UID: \"b6224b78-f7ed-435e-9283-5fcf886e4e5d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:50:59.956514 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:50:59.956426 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:51:00.101841 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:00.101816 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-wswqx"] Jul 16 13:51:00.102246 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:51:00.102224 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6224b78_f7ed_435e_9283_5fcf886e4e5d.slice/crio-7e329ebbd6e9dcaa933b8b17da493c95abf26461269f00c9b669966cc14e1ddd WatchSource:0}: Error finding container 7e329ebbd6e9dcaa933b8b17da493c95abf26461269f00c9b669966cc14e1ddd: Status 404 returned error can't find the container with id 7e329ebbd6e9dcaa933b8b17da493c95abf26461269f00c9b669966cc14e1ddd Jul 16 13:51:00.104299 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:00.104282 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:51:00.655043 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:00.655006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" event={"ID":"b6224b78-f7ed-435e-9283-5fcf886e4e5d","Type":"ContainerStarted","Data":"7e329ebbd6e9dcaa933b8b17da493c95abf26461269f00c9b669966cc14e1ddd"} Jul 16 13:51:02.675361 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:02.675325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" event={"ID":"b6224b78-f7ed-435e-9283-5fcf886e4e5d","Type":"ContainerStarted","Data":"db7f37aa0e7ed58b74901b842025f05abaa196580eb112fcbedaa9997773f9fe"} Jul 16 13:51:02.675785 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:02.675413 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:51:02.694546 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:02.694492 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" podStartSLOduration=2.1648687 podStartE2EDuration="3.694476646s" totalImagesPullingTime="1.529607946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:51:02.693272472 +0000 UTC m=+642.641389202" watchObservedRunningTime="2026-07-16 13:51:02.694476646 +0000 UTC m=+642.642593353" Jul 16 13:51:06.141080 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:06.141034 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6247 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 13:51:06.141556 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:06.141121 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podUID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 13:51:07.708272 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.708179 2563 generic.go:356] "Generic (PLEG): container finished" podID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerID="aee1b0e2e1ddea4cfb5b4444f055c6de4d552ee4b618f6e53a0276a065606e9f" exitCode=0 Jul 16 13:51:07.708272 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.708252 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" event={"ID":"0a0054dc-68ba-4482-a4b6-138f93c75a51","Type":"ContainerDied","Data":"aee1b0e2e1ddea4cfb5b4444f055c6de4d552ee4b618f6e53a0276a065606e9f"} Jul 16 13:51:07.708723 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.708624 2563 scope.go:117] "RemoveContainer" containerID="aee1b0e2e1ddea4cfb5b4444f055c6de4d552ee4b618f6e53a0276a065606e9f" Jul 16 13:51:07.709571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.709548 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:51:07.709644 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.709615 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:51:07.709694 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:07.709643 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 13:51:08.716029 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:08.715993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" event={"ID":"0a0054dc-68ba-4482-a4b6-138f93c75a51","Type":"ContainerStarted","Data":"b1007cafc45421b89d818955a2a56835507dc115b768a3d371116d6d2c2ec4a1"} Jul 16 13:51:08.921252 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:08.921209 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:51:08.921498 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:08.921473 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 16 13:51:08.921574 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:08.921524 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podUID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 16 13:51:09.214502 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:09.214452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:09.922092 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:09.922057 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 16 13:51:09.922494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:09.922117 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podUID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 16 13:51:10.921494 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:10.921451 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 16 13:51:10.921667 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:10.921519 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podUID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 16 13:51:11.705321 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:11.705283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:11.921717 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:11.921667 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 16 13:51:11.921934 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:11.921736 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" podUID="0a0054dc-68ba-4482-a4b6-138f93c75a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 16 13:51:12.925126 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:12.925096 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:51:12.925497 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:12.925373 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:51:12.926064 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:12.926043 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6247" Jul 16 13:51:13.684772 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:13.684739 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-wswqx" Jul 16 13:51:21.211124 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:21.211089 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:24.270278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:24.270242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:24.924405 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:24.924368 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:51:28.216129 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.216099 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz"] Jul 16 13:51:28.222045 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.222021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.226004 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.225983 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 13:51:28.226765 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.226747 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 13:51:28.228198 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.228182 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-krq85\"" Jul 16 13:51:28.228334 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.228309 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 13:51:28.252146 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.252109 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz"] Jul 16 13:51:28.316604 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj9d\" (UniqueName: \"kubernetes.io/projected/e21d7c36-a0fd-494a-a492-62e3d7bde22e-kube-api-access-klj9d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.316604 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.316987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.316987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.316987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316792 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.316987 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.316862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417571 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj9d\" (UniqueName: \"kubernetes.io/projected/e21d7c36-a0fd-494a-a492-62e3d7bde22e-kube-api-access-klj9d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.417767 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.417722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.418073 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.418050 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.418119 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.418091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.418179 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.418162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.419922 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.419905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21d7c36-a0fd-494a-a492-62e3d7bde22e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.420201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.420183 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21d7c36-a0fd-494a-a492-62e3d7bde22e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.427062 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.427037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj9d\" (UniqueName: \"kubernetes.io/projected/e21d7c36-a0fd-494a-a492-62e3d7bde22e-kube-api-access-klj9d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bddjz\" (UID: \"e21d7c36-a0fd-494a-a492-62e3d7bde22e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.532803 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.532771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:28.672354 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.672322 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz"] Jul 16 13:51:28.673556 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:51:28.673532 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21d7c36_a0fd_494a_a492_62e3d7bde22e.slice/crio-0a75638a0da014fecd649289a5dd7b12fdd4733feedf48941b68cdfc172c698d WatchSource:0}: Error finding container 0a75638a0da014fecd649289a5dd7b12fdd4733feedf48941b68cdfc172c698d: Status 404 returned error can't find the container with id 0a75638a0da014fecd649289a5dd7b12fdd4733feedf48941b68cdfc172c698d Jul 16 13:51:28.827201 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.827111 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" event={"ID":"e21d7c36-a0fd-494a-a492-62e3d7bde22e","Type":"ContainerStarted","Data":"0a75638a0da014fecd649289a5dd7b12fdd4733feedf48941b68cdfc172c698d"} Jul 16 13:51:28.906484 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:28.906442 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:51:29.487351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:29.487313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:30.839515 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:30.839478 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" event={"ID":"e21d7c36-a0fd-494a-a492-62e3d7bde22e","Type":"ContainerStarted","Data":"d17fa90a51e9593f5ae0d6212393eb57a42b03ca7b49580e32f4378c001eecc1"} Jul 16 13:51:30.882481 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:30.882422 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" podStartSLOduration=1.31711351 podStartE2EDuration="2.882403502s" totalImagesPullingTime="1.565289992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:51:30.881666576 +0000 UTC m=+670.829783277" watchObservedRunningTime="2026-07-16 13:51:30.882403502 +0000 UTC m=+670.830520210" Jul 16 13:51:30.899313 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:30.899275 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:32.217375 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:32.217333 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:51:38.532945 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:38.532909 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:38.533336 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:38.532971 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:38.546300 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:38.546276 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:38.909474 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:38.909388 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bddjz" Jul 16 13:51:40.192891 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.192847 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r"] Jul 16 13:51:40.196704 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.196687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.222167 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.222143 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 13:51:40.229886 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.229846 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r"] Jul 16 13:51:40.299826 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.299787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:51:40.331691 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331654 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f82r\" (UniqueName: \"kubernetes.io/projected/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-kube-api-access-8f82r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.331909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.331909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.331909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.331909 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.332097 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.331938 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.432634 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.432634 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.432908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.432908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.432908 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f82r\" (UniqueName: \"kubernetes.io/projected/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-kube-api-access-8f82r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.433083 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.432951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.433175 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.433097 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.433175 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.433148 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.433278 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.433160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.435175 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.435136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.435357 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.435342 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.450662 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.450596 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f82r\" (UniqueName: \"kubernetes.io/projected/ebdec5b6-6bdd-4b04-9aa0-7d91142605de-kube-api-access-8f82r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r\" (UID: \"ebdec5b6-6bdd-4b04-9aa0-7d91142605de\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.507384 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.507350 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:40.658845 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.658814 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r"] Jul 16 13:51:40.659037 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:51:40.659014 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdec5b6_6bdd_4b04_9aa0_7d91142605de.slice/crio-8ca29d8d00f419cdef93a99c05823006ba207e42ff474e0c6e585dd2444cdf2e WatchSource:0}: Error finding container 8ca29d8d00f419cdef93a99c05823006ba207e42ff474e0c6e585dd2444cdf2e: Status 404 returned error can't find the container with id 8ca29d8d00f419cdef93a99c05823006ba207e42ff474e0c6e585dd2444cdf2e Jul 16 13:51:40.909806 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:40.909773 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" event={"ID":"ebdec5b6-6bdd-4b04-9aa0-7d91142605de","Type":"ContainerStarted","Data":"8ca29d8d00f419cdef93a99c05823006ba207e42ff474e0c6e585dd2444cdf2e"} Jul 16 13:51:42.019143 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:42.019105 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:51:44.934345 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:44.934304 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" event={"ID":"ebdec5b6-6bdd-4b04-9aa0-7d91142605de","Type":"ContainerStarted","Data":"1af3ea60ca043471852edbe9d9bee2b0cc8b9aee9075dd298a50c949b8d5c26e"} Jul 16 13:51:44.962197 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:44.962144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" podStartSLOduration=1.121972723 podStartE2EDuration="4.962126021s" totalImagesPullingTime="3.840153298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:51:44.959637861 +0000 UTC m=+684.907754584" watchObservedRunningTime="2026-07-16 13:51:44.962126021 +0000 UTC m=+684.910242728" Jul 16 13:51:50.507554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:50.507516 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:50.507993 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:50.507610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:50.520200 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:50.520177 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:51:50.985692 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:51:50.985652 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r" Jul 16 13:52:20.215562 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:20.215446 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:52:20.319153 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:20.319114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:52:20.503805 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:20.503723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:52:22.516942 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:22.516909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:52:33.829488 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.829454 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4"] Jul 16 13:52:33.832111 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.832093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.834411 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.834387 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 13:52:33.848491 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.848455 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4"] Jul 16 13:52:33.871614 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.871573 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:52:33.929136 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929100 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.929136 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929141 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3dff74b-97dc-4b26-b59b-78737cca3392-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.929369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929160 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/a3dff74b-97dc-4b26-b59b-78737cca3392-kube-api-access-f7wg2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.929369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929251 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.929369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:33.929369 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:33.929353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030565 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030565 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3dff74b-97dc-4b26-b59b-78737cca3392-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030592 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/a3dff74b-97dc-4b26-b59b-78737cca3392-kube-api-access-f7wg2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.030816 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.030725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.031061 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.031009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.031125 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.031106 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.031231 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.031208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.033032 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.033010 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3dff74b-97dc-4b26-b59b-78737cca3392-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.033174 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.033155 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3dff74b-97dc-4b26-b59b-78737cca3392-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.038924 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.038899 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/a3dff74b-97dc-4b26-b59b-78737cca3392-kube-api-access-f7wg2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4\" (UID: \"a3dff74b-97dc-4b26-b59b-78737cca3392\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.141610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.141511 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:34.286610 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:34.286581 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4"] Jul 16 13:52:34.288324 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:52:34.288298 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3dff74b_97dc_4b26_b59b_78737cca3392.slice/crio-e8dfa799d295029061b5cfc09c60def9f0b30263a2053fcb67ddbd4fa3d065c1 WatchSource:0}: Error finding container e8dfa799d295029061b5cfc09c60def9f0b30263a2053fcb67ddbd4fa3d065c1: Status 404 returned error can't find the container with id e8dfa799d295029061b5cfc09c60def9f0b30263a2053fcb67ddbd4fa3d065c1 Jul 16 13:52:35.222144 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:35.222109 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" event={"ID":"a3dff74b-97dc-4b26-b59b-78737cca3392","Type":"ContainerStarted","Data":"97803387216eadc4e68d382b6ae737f5263fb238b860e5e4ea4f6e541b0ac7d5"} Jul 16 13:52:35.222554 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:35.222153 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" event={"ID":"a3dff74b-97dc-4b26-b59b-78737cca3392","Type":"ContainerStarted","Data":"e8dfa799d295029061b5cfc09c60def9f0b30263a2053fcb67ddbd4fa3d065c1"} Jul 16 13:52:35.245280 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:35.245219 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" podStartSLOduration=2.057032601 podStartE2EDuration="2.245202937s" totalImagesPullingTime="188.170336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:52:35.241174765 +0000 UTC m=+735.189291473" watchObservedRunningTime="2026-07-16 13:52:35.245202937 +0000 UTC m=+735.193319698" Jul 16 13:52:36.719013 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:36.718973 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:52:44.142631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:44.142582 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:44.142631 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:44.142640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:44.158391 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:44.158360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:44.293442 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:44.293409 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4" Jul 16 13:52:47.986741 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:47.986704 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:52:59.867435 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:52:59.867400 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:53:55.951562 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:53:55.951470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:53:58.055197 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:53:58.055166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:07.377733 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.377701 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-564fb575c6-jxlnd"] Jul 16 13:54:07.380408 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.380390 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.395199 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.395171 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564fb575c6-jxlnd"] Jul 16 13:54:07.397317 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.397295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4578272c-787f-420f-8a70-3d5e08fa9bea-tls-cert\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.397413 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.397380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpqq\" (UniqueName: \"kubernetes.io/projected/4578272c-787f-420f-8a70-3d5e08fa9bea-kube-api-access-wdpqq\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.498636 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.498600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpqq\" (UniqueName: \"kubernetes.io/projected/4578272c-787f-420f-8a70-3d5e08fa9bea-kube-api-access-wdpqq\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.498821 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.498706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4578272c-787f-420f-8a70-3d5e08fa9bea-tls-cert\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.501121 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.501101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4578272c-787f-420f-8a70-3d5e08fa9bea-tls-cert\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.509217 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.509193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpqq\" (UniqueName: \"kubernetes.io/projected/4578272c-787f-420f-8a70-3d5e08fa9bea-kube-api-access-wdpqq\") pod \"authorino-564fb575c6-jxlnd\" (UID: \"4578272c-787f-420f-8a70-3d5e08fa9bea\") " pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.690937 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.690829 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564fb575c6-jxlnd" Jul 16 13:54:07.856805 ip-10-0-130-14 kubenswrapper[2563]: W0716 13:54:07.856777 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4578272c_787f_420f_8a70_3d5e08fa9bea.slice/crio-1e3ca872bb145eee82c1a1065ce2102e198dcf34372c27888eaf4d0d2f9cc1bb WatchSource:0}: Error finding container 1e3ca872bb145eee82c1a1065ce2102e198dcf34372c27888eaf4d0d2f9cc1bb: Status 404 returned error can't find the container with id 1e3ca872bb145eee82c1a1065ce2102e198dcf34372c27888eaf4d0d2f9cc1bb Jul 16 13:54:07.862844 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.862800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564fb575c6-jxlnd"] Jul 16 13:54:07.871064 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.871036 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:07.997111 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:07.997014 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:08.754006 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:08.753906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564fb575c6-jxlnd" event={"ID":"4578272c-787f-420f-8a70-3d5e08fa9bea","Type":"ContainerStarted","Data":"a44b3603544433dcaf99ebe321794a0011be7176e368a8af74300e73096eba2b"} Jul 16 13:54:08.754006 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:08.753965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564fb575c6-jxlnd" event={"ID":"4578272c-787f-420f-8a70-3d5e08fa9bea","Type":"ContainerStarted","Data":"1e3ca872bb145eee82c1a1065ce2102e198dcf34372c27888eaf4d0d2f9cc1bb"} Jul 16 13:54:08.786236 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:08.786178 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-564fb575c6-jxlnd" podStartSLOduration=1.269455082 podStartE2EDuration="1.78616058s" totalImagesPullingTime="516.705498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:54:08.784385072 +0000 UTC m=+828.732501779" watchObservedRunningTime="2026-07-16 13:54:08.78616058 +0000 UTC m=+828.734277338" Jul 16 13:54:08.855286 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:08.855247 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:54:08.855723 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:08.855692 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d79d76977-ppxmt" podUID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" containerName="authorino" containerID="cri-o://d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90" gracePeriod=30 Jul 16 13:54:09.140609 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.140583 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:54:09.207700 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.207667 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:09.215766 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.215728 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d89pm\" (UniqueName: \"kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm\") pod \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " Jul 16 13:54:09.215954 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.215775 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert\") pod \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\" (UID: \"0066bed2-dcf0-4de3-a7f4-4feafaf90c60\") " Jul 16 13:54:09.218078 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.218048 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm" (OuterVolumeSpecName: "kube-api-access-d89pm") pod "0066bed2-dcf0-4de3-a7f4-4feafaf90c60" (UID: "0066bed2-dcf0-4de3-a7f4-4feafaf90c60"). InnerVolumeSpecName "kube-api-access-d89pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:54:09.227897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.227839 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0066bed2-dcf0-4de3-a7f4-4feafaf90c60" (UID: "0066bed2-dcf0-4de3-a7f4-4feafaf90c60"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:54:09.316686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.316594 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d89pm\" (UniqueName: \"kubernetes.io/projected/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-kube-api-access-d89pm\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:54:09.316686 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.316627 2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0066bed2-dcf0-4de3-a7f4-4feafaf90c60-tls-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 13:54:09.681905 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.681739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:09.762792 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.762756 2563 generic.go:356] "Generic (PLEG): container finished" podID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" containerID="d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90" exitCode=0 Jul 16 13:54:09.762792 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.762798 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d79d76977-ppxmt" Jul 16 13:54:09.763382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.762803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d79d76977-ppxmt" event={"ID":"0066bed2-dcf0-4de3-a7f4-4feafaf90c60","Type":"ContainerDied","Data":"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90"} Jul 16 13:54:09.763382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.762847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d79d76977-ppxmt" event={"ID":"0066bed2-dcf0-4de3-a7f4-4feafaf90c60","Type":"ContainerDied","Data":"e7fb376919b1612c0dd861c04747785fa097acdda6f4b6420148e56a2fa03403"} Jul 16 13:54:09.763382 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.762868 2563 scope.go:117] "RemoveContainer" containerID="d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90" Jul 16 13:54:09.774324 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.774306 2563 scope.go:117] "RemoveContainer" containerID="d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90" Jul 16 13:54:09.774620 ip-10-0-130-14 kubenswrapper[2563]: E0716 13:54:09.774603 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90\": container with ID starting with d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90 not found: ID does not exist" containerID="d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90" Jul 16 13:54:09.774710 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.774628 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90"} err="failed to get container status \"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90\": rpc error: code = NotFound desc = could not find container \"d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90\": container with ID starting with d52e2222ed8fed80b8e41169fe6e4f40a1b6e92592161beb6485f9acce34dc90 not found: ID does not exist" Jul 16 13:54:09.801792 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.801751 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:54:09.809014 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:09.808989 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d79d76977-ppxmt"] Jul 16 13:54:10.475405 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:10.475369 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" path="/var/lib/kubelet/pods/0066bed2-dcf0-4de3-a7f4-4feafaf90c60/volumes" Jul 16 13:54:25.527385 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:25.527350 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:54:45.220533 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:45.220496 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:54:54.924648 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:54:54.924597 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:55:05.317040 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:05.317000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:55:14.643897 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:14.643798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:55:20.673309 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:20.673281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:55:20.674790 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:20.674765 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 13:55:26.524795 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:26.524762 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:55:27.884048 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:27.884008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:55:29.356404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:29.356370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:55:51.813757 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:55:51.813717 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:56:06.107049 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:06.107016 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:56:11.619381 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:11.619341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:13.525293 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:13.525261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:18.229151 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:18.229120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:18.307350 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:18.307312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:18.425890 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:18.425835 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:18.523860 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:18.523828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:18.660506 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:18.660465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:25.261112 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:25.261075 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:25.366404 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:25.366371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:56:44.295028 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:44.294944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:56:52.206134 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:56:52.206097 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:02.804337 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:02.804298 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:57:03.001957 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:03.001919 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:11.414003 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:11.409129 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:22.903675 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:22.903629 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:34.202351 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:34.202320 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:43.749386 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:43.749354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:57:43.849049 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:43.849009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 13:57:45.402292 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:45.402254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:57:55.202085 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:57:55.202046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:58:21.205868 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:58:21.205770 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:58:33.093629 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:58:33.093595 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:58:55.002941 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:58:55.002860 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:59:05.910229 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:59:05.910195 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:59:23.801527 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:59:23.801490 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:59:33.401095 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:59:33.401057 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:59:43.203907 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:59:43.203798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 13:59:52.701698 ip-10-0-130-14 kubenswrapper[2563]: I0716 13:59:52.701652 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:00:03.904331 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:03.904291 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:00:20.711930 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:20.711902 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:00:20.715447 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:20.715420 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:00:26.097564 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:26.097528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:00:28.789296 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:28.789256 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 14:00:28.789768 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:28.789510 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" podUID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" containerName="manager" containerID="cri-o://491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7" gracePeriod=10 Jul 16 14:00:32.759845 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.759818 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 14:00:32.947075 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.946986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t5zn\" (UniqueName: \"kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn\") pod \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " Jul 16 14:00:32.947075 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.947026 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume\") pod \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\" (UID: \"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585\") " Jul 16 14:00:32.947422 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.947399 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" (UID: "e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:00:32.949090 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.949068 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn" (OuterVolumeSpecName: "kube-api-access-2t5zn") pod "e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" (UID: "e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585"). InnerVolumeSpecName "kube-api-access-2t5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:00:32.993959 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.993922 2563 generic.go:356] "Generic (PLEG): container finished" podID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" containerID="491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7" exitCode=0 Jul 16 14:00:32.994155 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.993974 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" Jul 16 14:00:32.994155 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.994016 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" event={"ID":"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585","Type":"ContainerDied","Data":"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7"} Jul 16 14:00:32.994155 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.994083 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k" event={"ID":"e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585","Type":"ContainerDied","Data":"bcef479d0e9b18cde12d2a0a74272085916066ab29855c3957ef930eacf26c1f"} Jul 16 14:00:32.994155 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:32.994112 2563 scope.go:117] "RemoveContainer" containerID="491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7" Jul 16 14:00:33.004730 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.004709 2563 scope.go:117] "RemoveContainer" containerID="491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7" Jul 16 14:00:33.005021 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:00:33.004999 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7\": container with ID starting with 491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7 not found: ID does not exist" containerID="491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7" Jul 16 14:00:33.005078 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.005030 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7"} err="failed to get container status \"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7\": rpc error: code = NotFound desc = could not find container \"491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7\": container with ID starting with 491637f8b7c480b11dc99633e4972c94292da7807dc51d34f98fae2d8a14b2d7 not found: ID does not exist" Jul 16 14:00:33.026142 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.026112 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 14:00:33.033231 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.033204 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdb9k"] Jul 16 14:00:33.048329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.048301 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5zn\" (UniqueName: \"kubernetes.io/projected/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-kube-api-access-2t5zn\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:00:33.048329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:33.048329 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585-extensions-socket-volume\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:00:34.474799 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:00:34.474765 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" path="/var/lib/kubelet/pods/e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585/volumes" Jul 16 14:01:05.021379 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:05.021342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:01:30.027259 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.027218 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs"] Jul 16 14:01:30.030266 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.030236 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" containerName="authorino" Jul 16 14:01:30.030457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.030445 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" containerName="authorino" Jul 16 14:01:30.030587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.030577 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" containerName="manager" Jul 16 14:01:30.030663 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.030655 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" containerName="manager" Jul 16 14:01:30.031715 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.031694 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b8ffa8-839e-4e69-a7d0-fb4d1e6ff585" containerName="manager" Jul 16 14:01:30.031842 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.031831 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0066bed2-dcf0-4de3-a7f4-4feafaf90c60" containerName="authorino" Jul 16 14:01:30.037787 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.037762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.040863 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.040413 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5pqhk\"" Jul 16 14:01:30.043248 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.043226 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs"] Jul 16 14:01:30.068667 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.068632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d6c259d-1f12-4506-a925-0d1e77039971-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.068858 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.068686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5w6\" (UniqueName: \"kubernetes.io/projected/3d6c259d-1f12-4506-a925-0d1e77039971-kube-api-access-jj5w6\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.169644 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.169609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d6c259d-1f12-4506-a925-0d1e77039971-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.169832 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.169677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5w6\" (UniqueName: \"kubernetes.io/projected/3d6c259d-1f12-4506-a925-0d1e77039971-kube-api-access-jj5w6\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.170150 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.170123 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d6c259d-1f12-4506-a925-0d1e77039971-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.179380 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.179352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5w6\" (UniqueName: \"kubernetes.io/projected/3d6c259d-1f12-4506-a925-0d1e77039971-kube-api-access-jj5w6\") pod \"kuadrant-operator-controller-manager-86878c769f-q67hs\" (UID: \"3d6c259d-1f12-4506-a925-0d1e77039971\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.350501 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.350406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:30.514692 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:30.514662 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs"] Jul 16 14:01:30.519322 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:01:30.519291 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6c259d_1f12_4506_a925_0d1e77039971.slice/crio-a6ac40aaf8c6212a7220c4aa9f37b7e06750644a23ac0f2860517a5864e93c6c WatchSource:0}: Error finding container a6ac40aaf8c6212a7220c4aa9f37b7e06750644a23ac0f2860517a5864e93c6c: Status 404 returned error can't find the container with id a6ac40aaf8c6212a7220c4aa9f37b7e06750644a23ac0f2860517a5864e93c6c Jul 16 14:01:31.317088 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:31.317048 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" event={"ID":"3d6c259d-1f12-4506-a925-0d1e77039971","Type":"ContainerStarted","Data":"79cbae447ed6f65093dce47b26722ebb7844796e2331723ab83c6b28d36fd3c4"} Jul 16 14:01:31.317502 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:31.317119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:31.317502 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:31.317142 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" event={"ID":"3d6c259d-1f12-4506-a925-0d1e77039971","Type":"ContainerStarted","Data":"a6ac40aaf8c6212a7220c4aa9f37b7e06750644a23ac0f2860517a5864e93c6c"} Jul 16 14:01:31.343149 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:31.343091 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" podStartSLOduration=1.343075628 podStartE2EDuration="1.343075628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:01:31.34215433 +0000 UTC m=+1271.290271062" watchObservedRunningTime="2026-07-16 14:01:31.343075628 +0000 UTC m=+1271.291192334" Jul 16 14:01:42.325584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:42.325555 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q67hs" Jul 16 14:01:51.193331 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:01:51.193288 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:00.701880 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:00.701820 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:13.899985 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:13.899940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:26.903016 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:26.902982 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:37.406234 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:37.406194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:47.404114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:47.404030 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:02:56.910095 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:02:56.910060 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:03:01.824083 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:01.824051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:03:06.996748 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:06.996711 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:03:18.197910 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:18.197851 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:03:25.399445 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:25.399406 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:03:35.615919 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:35.615865 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:03:44.909839 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:03:44.909788 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:04:05.305906 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:04:05.305854 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:04:13.001229 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:04:13.001139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:04:22.627948 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:04:22.627913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:04:35.811763 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:04:35.811728 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:04:52.497964 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:04:52.497923 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:05:05.198514 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:05.198470 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:05:14.510842 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:14.510803 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:05:20.754562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:20.754528 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:05:20.758857 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:20.758834 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:05:28.707855 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:28.707797 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:05:38.593967 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:38.593928 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:05:52.298487 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:05:52.298404 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:06.902082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:06.902045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:17.102849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:17.102805 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:23.700178 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:23.700138 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:28.614036 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:28.614002 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:06:28.703143 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:28.703087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:06:34.908021 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:34.907983 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:45.710983 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:45.710949 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:06:58.604234 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:06:58.604193 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:10.612441 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:10.612401 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:20.402094 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:20.402018 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:23.109010 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:23.108969 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:33.033317 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:33.033284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:33.500826 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:33.500744 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:41.329600 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:41.329550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:42.098221 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:42.098191 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:46.531148 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:46.531111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:48.181538 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:48.181493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:49.865452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:49.865382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:49.963477 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:49.963423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:50.319092 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:50.319062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:50.709238 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:50.709155 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:07:56.345150 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:56.345089 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:56.445821 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:56.445773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:57.289940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:57.289903 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:57.390229 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:57.390203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:07:57.491066 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:07:57.491035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:00.198192 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:00.198153 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:08:00.727745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:00.727696 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:03.617383 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:03.617356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:03.715604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:03.715575 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:06.956503 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:06.956471 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:09.398018 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:09.397977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:08:09.705799 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:09.705713 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:09.938681 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:09.938648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:10.002231 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:10.002148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:10.214584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:10.214554 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:10.317982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:10.317951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:10.943219 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:10.943187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:11.044470 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:11.044441 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:11.273667 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:11.273632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:11.374271 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:11.374223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:14.237298 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:14.237249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:16.402832 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:16.402804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:16.505218 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:16.505188 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:18.406583 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:18.406544 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:22.570777 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:22.570743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:22.635559 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:22.635527 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:24.269115 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:24.269078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:24.382105 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:24.382070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:26.345787 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:26.345678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:26.791655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:26.791622 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:08:28.519485 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:28.519446 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:28.631156 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:28.631126 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:30.119910 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:30.119853 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:34.704085 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:34.704045 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:34.781945 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:34.781904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:35.314459 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:35.314426 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:35.418086 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:35.418059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:36.404019 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:36.403980 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:08:36.989426 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:36.989394 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:43.040381 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:43.040294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:43.147996 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:43.147962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:43.242117 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:43.242080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:43.342134 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:43.342038 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:08:43.864529 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:08:43.864496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:09.906142 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:09.906099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:15.765121 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:15.765083 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:15.866379 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:15.866348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:15.966250 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:15.966203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:31.951834 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:31.951792 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:38.034307 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:38.034262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:38.146717 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:38.146679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:38.293638 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:38.293541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:48.859303 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:48.859266 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:49.698346 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:49.698309 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:55.382548 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:55.382510 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:55.450678 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:55.450642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:55.554118 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:55.554084 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:55.642963 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:55.642865 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:55.743102 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:55.743066 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:56.582591 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:56.582550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:56.680765 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:56.680737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:56.780745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:56.780713 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:09:57.998109 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:57.998078 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:09:58.521731 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:09:58.521697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:04.518251 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:04.518160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:04.580986 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:04.580951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:05.265930 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:05.265866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:07.401200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:07.401160 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:10:11.078938 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.078864 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:11.145495 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.145463 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:11.653863 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.653830 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:11.659135 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.659103 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.662069 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.662041 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-49f7930e-openshift-default-dockercfg-pgb5g\"" Jul 16 14:10:11.681460 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.681426 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:11.772532 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772714 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772539 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772714 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772714 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772714 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qsx\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772859 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772859 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772859 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.772859 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.772830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873702 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873731 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qsx\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873790 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.873949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.873914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.874322 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.874162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.874412 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.874387 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.874473 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.874416 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.874530 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.874512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.874583 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.874512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.876558 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.876529 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.876744 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.876727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.882353 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.882327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.882495 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.882477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qsx\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx\") pod \"e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:11.970726 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:11.970621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:12.131715 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.131684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:12.133082 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:10:12.133052 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2e4000_b90a_4f45_ae57_c52a638936e8.slice/crio-1a07ccbad453177247628d8b75b558a79d8187a6537093835711c88d9751272e WatchSource:0}: Error finding container 1a07ccbad453177247628d8b75b558a79d8187a6537093835711c88d9751272e: Status 404 returned error can't find the container with id 1a07ccbad453177247628d8b75b558a79d8187a6537093835711c88d9751272e Jul 16 14:10:12.135286 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.135253 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:12.135368 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.135324 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:12.135368 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.135353 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:12.415965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.415927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerStarted","Data":"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42"} Jul 16 14:10:12.416170 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.415986 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerStarted","Data":"1a07ccbad453177247628d8b75b558a79d8187a6537093835711c88d9751272e"} Jul 16 14:10:12.435668 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.435615 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podStartSLOduration=1.435592441 podStartE2EDuration="1.435592441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:10:12.435161472 +0000 UTC m=+1792.383278190" watchObservedRunningTime="2026-07-16 14:10:12.435592441 +0000 UTC m=+1792.383709149" Jul 16 14:10:12.481253 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.481209 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:12.535570 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.535530 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:12.971190 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:12.971157 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:13.648025 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:13.647989 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:13.750583 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:13.750547 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:13.865489 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:13.865455 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:13.969949 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:13.969816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:13.976773 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:13.976742 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:14.364183 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:14.364147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:14.428156 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:14.428120 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:14.429096 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:14.429080 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:14.561309 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:14.561265 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:19.223476 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:19.223444 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:19.978750 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:19.978716 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:20.079345 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:20.079310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:20.405321 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:20.405286 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:10:20.512337 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:20.512302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:20.794651 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:20.794612 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:10:20.808914 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:20.808867 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:10:26.373678 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:26.373639 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:26.605319 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:26.605279 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:27.122243 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:27.122200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:29.429350 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.429315 2563 patch_prober.go:28] interesting pod/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 16 14:10:29.429808 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.429382 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 16 14:10:29.517403 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.517368 2563 generic.go:356] "Generic (PLEG): container finished" podID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerID="430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42" exitCode=0 Jul 16 14:10:29.517403 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.517407 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerDied","Data":"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42"} Jul 16 14:10:29.517827 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.517812 2563 scope.go:117] "RemoveContainer" containerID="430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42" Jul 16 14:10:29.518920 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.518886 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:29.519070 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.518979 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:29.519070 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:29.519033 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:10:30.525585 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:30.525550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerStarted","Data":"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315"} Jul 16 14:10:30.971655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:30.971552 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:30.971962 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:30.971921 2563 patch_prober.go:28] interesting pod/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 16 14:10:30.972122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:30.971976 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 16 14:10:31.971279 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:31.971234 2563 patch_prober.go:28] interesting pod/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 16 14:10:31.971680 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:31.971338 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 16 14:10:32.971335 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:32.971295 2563 patch_prober.go:28] interesting pod/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 16 14:10:32.971742 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:32.971395 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 16 14:10:32.999819 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:32.999790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.100589 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.100554 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.281899 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.281225 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:33.281899 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.281463 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" containerID="cri-o://6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315" gracePeriod=30 Jul 16 14:10:33.291481 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.291447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.535384 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.535287 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.666855 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.666817 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.766503 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.766454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:33.913900 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:33.913783 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:36.797290 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:36.797260 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:10:39.962953 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:39.962919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:40.076144 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.076106 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:40.168255 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.168204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:40.335383 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.335355 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:40.343420 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.343390 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:40.458312 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458279 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458312 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458318 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458390 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458471 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458497 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458536 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458562 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458587 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7qsx\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx\") pod \"2a2e4000-b90a-4f45-ae57-c52a638936e8\" (UID: \"2a2e4000-b90a-4f45-ae57-c52a638936e8\") " Jul 16 14:10:40.458973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458672 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:10:40.458973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458707 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 14:10:40.458973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458864 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:10:40.458973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458943 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2e4000-b90a-4f45-ae57-c52a638936e8-istiod-ca-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.458973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.458962 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.459216 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.459151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:10:40.459457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.459431 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:10:40.460833 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.460798 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:10:40.460833 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.460818 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 14:10:40.461329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.461295 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx" (OuterVolumeSpecName: "kube-api-access-p7qsx") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "kube-api-access-p7qsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:10:40.461329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.461313 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a2e4000-b90a-4f45-ae57-c52a638936e8" (UID: "2a2e4000-b90a-4f45-ae57-c52a638936e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:10:40.560045 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560002 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-podinfo\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560045 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560040 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560045 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560053 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-envoy\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560299 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560065 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-credential-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560299 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560078 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-workload-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560299 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560089 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2e4000-b90a-4f45-ae57-c52a638936e8-istio-data\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.560299 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.560100 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7qsx\" (UniqueName: \"kubernetes.io/projected/2a2e4000-b90a-4f45-ae57-c52a638936e8-kube-api-access-p7qsx\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:10:40.604713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.604681 2563 generic.go:356] "Generic (PLEG): container finished" podID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerID="6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315" exitCode=0 Jul 16 14:10:40.604940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.604793 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" Jul 16 14:10:40.604940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.604821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerDied","Data":"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315"} Jul 16 14:10:40.604940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.604894 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv" event={"ID":"2a2e4000-b90a-4f45-ae57-c52a638936e8","Type":"ContainerDied","Data":"1a07ccbad453177247628d8b75b558a79d8187a6537093835711c88d9751272e"} Jul 16 14:10:40.604940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.604924 2563 scope.go:117] "RemoveContainer" containerID="6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315" Jul 16 14:10:40.615144 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.615124 2563 scope.go:117] "RemoveContainer" containerID="430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42" Jul 16 14:10:40.624040 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.624014 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:40.624309 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.624294 2563 scope.go:117] "RemoveContainer" containerID="6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315" Jul 16 14:10:40.624598 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:10:40.624577 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315\": container with ID starting with 6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315 not found: ID does not exist" containerID="6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315" Jul 16 14:10:40.624657 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.624608 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315"} err="failed to get container status \"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315\": rpc error: code = NotFound desc = could not find container \"6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315\": container with ID starting with 6d3281c3ada53f358bca142e139b0765abd07ed6921a5614a3ae0540d9224315 not found: ID does not exist" Jul 16 14:10:40.624657 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.624635 2563 scope.go:117] "RemoveContainer" containerID="430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42" Jul 16 14:10:40.624941 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:10:40.624920 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42\": container with ID starting with 430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42 not found: ID does not exist" containerID="430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42" Jul 16 14:10:40.624994 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.624949 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42"} err="failed to get container status \"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42\": rpc error: code = NotFound desc = could not find container \"430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42\": container with ID starting with 430df2ca3951f34e97c9f62a1678d16fa8544df1980eea0f741187bfba8a4a42 not found: ID does not exist" Jul 16 14:10:40.631141 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:40.631116 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-49f7930e-openshift-default-5cbcc45fc9-w6mzv"] Jul 16 14:10:42.475554 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:42.475523 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" path="/var/lib/kubelet/pods/2a2e4000-b90a-4f45-ae57-c52a638936e8/volumes" Jul 16 14:10:47.107340 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:47.107303 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:47.206122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:47.206091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:50.619335 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:50.619300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:50.722854 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:50.722823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:54.180551 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:54.180518 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:54.281457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:54.281418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:54.382124 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:54.382090 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:55.248170 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:55.248105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:55.348153 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:55.348098 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:55.446134 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:55.446101 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:57.030825 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:57.030780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:10:58.712479 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:10:58.712440 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:11:03.018364 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:03.018328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:03.164068 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:03.164033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:03.735177 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:03.735144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:10.044522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:10.044487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:10.153679 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:10.153595 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:10.272092 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:10.272052 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:12.209526 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:12.209484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:11:20.902194 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:20.902157 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:21.291111 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:21.291068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:42.354401 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:42.354313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:48.762106 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:48.762057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:48.862283 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:48.862249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:49.150927 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.150852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:49.495404 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495320 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:11:49.495789 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495776 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:11:49.495839 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495791 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:11:49.495839 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495803 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:11:49.495839 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495809 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:11:49.495954 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.495918 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:11:49.498131 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.498116 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.501546 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.501527 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6cdc6b-openshift-default-dockercfg-dlgrg\"" Jul 16 14:11:49.512319 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.512287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:11:49.515586 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.515561 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:49.605565 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.605525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.605779 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.605585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.605779 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.605756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtlr\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.605946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.605815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.605946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.605851 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.606053 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.606010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.606099 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.606060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.606154 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.606136 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.606191 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.606170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.643945 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.643915 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:49.707239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtlr\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707522 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707442 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707905 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.707905 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707870 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.708027 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.707992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.708189 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.708149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.708267 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.708233 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.709746 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.709727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.709838 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.709820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.715748 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.715719 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.715854 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.715796 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtlr\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr\") pod \"e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.808589 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.808548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:49.957993 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.957959 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:11:49.959422 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:11:49.959350 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f672d71_f00d_47a9_8158_8c166c5cf1e7.slice/crio-82a41dff6d320fe1834f0e19be0ae996a9e62526f9b203800e187f4799a40262 WatchSource:0}: Error finding container 82a41dff6d320fe1834f0e19be0ae996a9e62526f9b203800e187f4799a40262: Status 404 returned error can't find the container with id 82a41dff6d320fe1834f0e19be0ae996a9e62526f9b203800e187f4799a40262 Jul 16 14:11:49.961803 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.961767 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:11:49.961908 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.961839 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:11:49.961908 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:49.961889 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:11:50.009979 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:50.009945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerStarted","Data":"82a41dff6d320fe1834f0e19be0ae996a9e62526f9b203800e187f4799a40262"} Jul 16 14:11:51.018149 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:51.018104 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerStarted","Data":"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1"} Jul 16 14:11:51.052404 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:51.052352 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podStartSLOduration=2.052335023 podStartE2EDuration="2.052335023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:11:51.050210941 +0000 UTC m=+1890.998327675" watchObservedRunningTime="2026-07-16 14:11:51.052335023 +0000 UTC m=+1891.000451729" Jul 16 14:11:51.809239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:51.809199 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:51.814161 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:51.814136 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:52.024517 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:52.024486 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:52.025366 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:52.025350 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:11:52.189466 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:52.189384 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:52.352120 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:52.352087 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:52.530386 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:52.530358 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:53.149862 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:53.149820 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:53.242721 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:53.242691 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:57.712900 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:57.712834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:58.099644 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:58.099593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:58.454106 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:58.454009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:11:59.385683 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:11:59.385647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:00.855520 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:00.855478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:00.979942 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:00.979886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:01.075213 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:01.075170 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:01.193269 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:01.193172 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:01.207239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:01.207198 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:12:01.607383 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:01.607343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:01.711515 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:01.711466 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:06.394359 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:06.394313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:08.066047 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:08.066006 2563 patch_prober.go:28] interesting pod/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 14:12:08.066475 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:08.066099 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 14:12:10.135415 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.135379 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerID="594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1" exitCode=0 Jul 16 14:12:10.135886 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.135452 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerDied","Data":"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1"} Jul 16 14:12:10.135886 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.135858 2563 scope.go:117] "RemoveContainer" containerID="594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1" Jul 16 14:12:10.136851 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.136826 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:12:10.136940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.136925 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:12:10.136983 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:10.136969 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:12:11.145256 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:11.145215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerStarted","Data":"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421"} Jul 16 14:12:11.809435 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:11.809396 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:12:11.809654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:11.809630 2563 patch_prober.go:28] interesting pod/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 16 14:12:11.809719 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:11.809689 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 16 14:12:12.809864 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:12.809825 2563 patch_prober.go:28] interesting pod/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 16 14:12:12.810289 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:12.809924 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 16 14:12:13.809364 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:13.809325 2563 patch_prober.go:28] interesting pod/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 16 14:12:13.809558 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:13.809382 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 16 14:12:14.161667 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:14.161570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:14.262329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:14.262285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:14.809497 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:14.809457 2563 patch_prober.go:28] interesting pod/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 16 14:12:14.809705 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:14.809512 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 16 14:12:14.814047 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:14.814021 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:12:15.812933 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:15.812904 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:12:15.813305 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:15.813239 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:12:15.813928 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:15.813912 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:12:17.225154 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.225125 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:12:17.225636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.225622 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2e4000-b90a-4f45-ae57-c52a638936e8" containerName="istio-proxy" Jul 16 14:12:17.229347 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.229324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.231818 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.231798 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6cdc6b\"/\"kube-root-ca.crt\"" Jul 16 14:12:17.232574 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.232558 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6cdc6b\"/\"test-model-6cdc6b-kserve-self-signed-certs\"" Jul 16 14:12:17.232648 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.232558 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6cdc6b\"/\"default-dockercfg-lgmtd\"" Jul 16 14:12:17.232648 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.232559 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6cdc6b\"/\"openshift-service-ca.crt\"" Jul 16 14:12:17.247409 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.247353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:12:17.263745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.263711 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:17.387345 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387307 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.387345 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.387634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.387634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.387634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndq6\" (UniqueName: \"kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.387782 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.387672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489063 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.488956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489063 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndq6\" (UniqueName: \"kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489302 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489302 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489302 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489302 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489521 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489521 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489517 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.489634 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.489596 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.491339 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.491313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.491734 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.491713 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.502334 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.502309 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndq6\" (UniqueName: \"kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6\") pod \"test-model-6cdc6b-kserve-647b6fbd85-w6lnr\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.550034 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.549998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:17.694925 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:17.694894 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:12:17.696286 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:12:17.696261 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d10715e_0c52_494b_8881_acb9ea5e67f6.slice/crio-7586b8bcf1619c3a5530cc4cb38d1272a0b4e7fe8ce48de04a80827b2ff85454 WatchSource:0}: Error finding container 7586b8bcf1619c3a5530cc4cb38d1272a0b4e7fe8ce48de04a80827b2ff85454: Status 404 returned error can't find the container with id 7586b8bcf1619c3a5530cc4cb38d1272a0b4e7fe8ce48de04a80827b2ff85454 Jul 16 14:12:18.195499 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:18.195463 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" event={"ID":"6d10715e-0c52-494b-8881-acb9ea5e67f6","Type":"ContainerStarted","Data":"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719"} Jul 16 14:12:18.195723 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:18.195513 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" event={"ID":"6d10715e-0c52-494b-8881-acb9ea5e67f6","Type":"ContainerStarted","Data":"7586b8bcf1619c3a5530cc4cb38d1272a0b4e7fe8ce48de04a80827b2ff85454"} Jul 16 14:12:18.216055 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:18.216006 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" podStartSLOduration=1.215989657 podStartE2EDuration="1.215989657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:12:18.21458201 +0000 UTC m=+1918.162698730" watchObservedRunningTime="2026-07-16 14:12:18.215989657 +0000 UTC m=+1918.164106364" Jul 16 14:12:23.449489 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:23.449453 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:23.549689 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:23.549651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:27.550206 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:27.550165 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:27.550649 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:27.550225 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:27.563008 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:27.562976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:27.800505 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:27.800420 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:12:28.269320 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:28.269288 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:12:41.877291 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:41.877229 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:12:47.798149 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:12:47.798109 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:13:05.455197 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:05.455161 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:13:05.493856 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:05.493820 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:06.449186 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:06.449145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:07.313297 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:07.313253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:12.049731 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:12.049615 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:14.295720 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:14.295683 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:13:20.480379 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.480308 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" podUID="6d10715e-0c52-494b-8881-acb9ea5e67f6" containerName="main" containerID="cri-o://e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719" gracePeriod=45 Jul 16 14:13:20.737047 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.736978 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:13:20.805215 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tndq6\" (UniqueName: \"kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805264 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805306 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805325 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805373 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805410 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs\") pod \"6d10715e-0c52-494b-8881-acb9ea5e67f6\" (UID: \"6d10715e-0c52-494b-8881-acb9ea5e67f6\") " Jul 16 14:13:20.805719 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:20.805719 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805649 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home" (OuterVolumeSpecName: "home") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:20.805719 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.805680 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:20.807529 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.807505 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6" (OuterVolumeSpecName: "kube-api-access-tndq6") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "kube-api-access-tndq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:13:20.807653 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.807520 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 14:13:20.807653 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.807531 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm" (OuterVolumeSpecName: "dshm") pod "6d10715e-0c52-494b-8881-acb9ea5e67f6" (UID: "6d10715e-0c52-494b-8881-acb9ea5e67f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:20.906147 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906105 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-tmp-dir\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:20.906147 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906140 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d10715e-0c52-494b-8881-acb9ea5e67f6-tls-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:20.906147 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906155 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndq6\" (UniqueName: \"kubernetes.io/projected/6d10715e-0c52-494b-8881-acb9ea5e67f6-kube-api-access-tndq6\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:20.906398 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906167 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-home\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:20.906398 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906180 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-model-cache\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:20.906398 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:20.906190 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d10715e-0c52-494b-8881-acb9ea5e67f6-dshm\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:21.583676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.583639 2563 generic.go:356] "Generic (PLEG): container finished" podID="6d10715e-0c52-494b-8881-acb9ea5e67f6" containerID="e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719" exitCode=0 Jul 16 14:13:21.584136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.583686 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" Jul 16 14:13:21.584136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.583722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" event={"ID":"6d10715e-0c52-494b-8881-acb9ea5e67f6","Type":"ContainerDied","Data":"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719"} Jul 16 14:13:21.584136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.583769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr" event={"ID":"6d10715e-0c52-494b-8881-acb9ea5e67f6","Type":"ContainerDied","Data":"7586b8bcf1619c3a5530cc4cb38d1272a0b4e7fe8ce48de04a80827b2ff85454"} Jul 16 14:13:21.584136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.583787 2563 scope.go:117] "RemoveContainer" containerID="e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719" Jul 16 14:13:21.593341 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.593321 2563 scope.go:117] "RemoveContainer" containerID="e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719" Jul 16 14:13:21.593602 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:13:21.593573 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719\": container with ID starting with e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719 not found: ID does not exist" containerID="e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719" Jul 16 14:13:21.593704 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.593602 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719"} err="failed to get container status \"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719\": rpc error: code = NotFound desc = could not find container \"e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719\": container with ID starting with e506dd10ecc776f21da3d813f77972db424e794345c411bb9dd119f3154dc719 not found: ID does not exist" Jul 16 14:13:21.605501 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.605475 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:13:21.609544 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:21.609521 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6cdc6b/test-model-6cdc6b-kserve-647b6fbd85-w6lnr"] Jul 16 14:13:22.474814 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:22.474780 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d10715e-0c52-494b-8881-acb9ea5e67f6" path="/var/lib/kubelet/pods/6d10715e-0c52-494b-8881-acb9ea5e67f6/volumes" Jul 16 14:13:48.987212 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:48.987171 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:49.275996 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:49.275962 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:13:49.276222 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:49.276178 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" containerID="cri-o://d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421" gracePeriod=30 Jul 16 14:13:49.307161 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:49.307120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:49.541310 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:49.541216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:50.068548 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:50.068512 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:52.309073 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:52.309036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:13:54.543004 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.542938 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:13:54.626770 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626733 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.626770 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626800 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626825 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626857 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626907 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chtlr\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.626953 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627006 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627057 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627059 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data\") pod \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\" (UID: \"4f672d71-f00d-47a9-8158-8c166c5cf1e7\") " Jul 16 14:13:54.627403 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627251 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:54.627403 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627300 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 14:13:54.627504 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627455 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istiod-ca-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.627504 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627477 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.627618 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:54.627675 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.627641 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:54.628135 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.628111 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:54.629161 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.629133 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:13:54.629273 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.629255 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:13:54.629450 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.629428 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 14:13:54.629533 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.629510 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr" (OuterVolumeSpecName: "kube-api-access-chtlr") pod "4f672d71-f00d-47a9-8158-8c166c5cf1e7" (UID: "4f672d71-f00d-47a9-8158-8c166c5cf1e7"). InnerVolumeSpecName "kube-api-access-chtlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:13:54.728136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728102 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-data\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728131 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-podinfo\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728141 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728149 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-workload-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728408 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728158 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chtlr\" (UniqueName: \"kubernetes.io/projected/4f672d71-f00d-47a9-8158-8c166c5cf1e7-kube-api-access-chtlr\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728408 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728167 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-istio-envoy\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.728408 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.728175 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f672d71-f00d-47a9-8158-8c166c5cf1e7-credential-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:13:54.778454 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.778423 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerID="d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421" exitCode=0 Jul 16 14:13:54.778629 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.778504 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" Jul 16 14:13:54.778629 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.778514 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerDied","Data":"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421"} Jul 16 14:13:54.778629 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.778575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz" event={"ID":"4f672d71-f00d-47a9-8158-8c166c5cf1e7","Type":"ContainerDied","Data":"82a41dff6d320fe1834f0e19be0ae996a9e62526f9b203800e187f4799a40262"} Jul 16 14:13:54.778629 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.778589 2563 scope.go:117] "RemoveContainer" containerID="d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421" Jul 16 14:13:54.790300 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.790283 2563 scope.go:117] "RemoveContainer" containerID="594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1" Jul 16 14:13:54.800262 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.800229 2563 scope.go:117] "RemoveContainer" containerID="d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421" Jul 16 14:13:54.800600 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:13:54.800582 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421\": container with ID starting with d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421 not found: ID does not exist" containerID="d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421" Jul 16 14:13:54.800676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.800612 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421"} err="failed to get container status \"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421\": rpc error: code = NotFound desc = could not find container \"d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421\": container with ID starting with d05558edd7e35a86da757be4f1f6c681fc7373c2e902e0f339388bf021570421 not found: ID does not exist" Jul 16 14:13:54.800676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.800638 2563 scope.go:117] "RemoveContainer" containerID="594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1" Jul 16 14:13:54.801040 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:13:54.801021 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1\": container with ID starting with 594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1 not found: ID does not exist" containerID="594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1" Jul 16 14:13:54.801098 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.801047 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1"} err="failed to get container status \"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1\": rpc error: code = NotFound desc = could not find container \"594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1\": container with ID starting with 594ea4dd69baa0cf505073c401b6c1f719bbc78bf6a5cb9d6466a508250a5bf1 not found: ID does not exist" Jul 16 14:13:54.803239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.803209 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:13:54.806120 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:54.806100 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6cdc6b-openshift-default-79c6f6876f-jxdkz"] Jul 16 14:13:56.272106 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:56.272068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:56.372896 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:56.372844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:56.474854 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:56.474823 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" path="/var/lib/kubelet/pods/4f672d71-f00d-47a9-8158-8c166c5cf1e7/volumes" Jul 16 14:13:56.534783 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:56.534678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:56.764502 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:56.764471 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:57.159987 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.159949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:57.556925 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.556860 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:13:57.557570 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557551 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:13:57.557713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557573 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:13:57.557713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557587 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d10715e-0c52-494b-8881-acb9ea5e67f6" containerName="main" Jul 16 14:13:57.557713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557596 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d10715e-0c52-494b-8881-acb9ea5e67f6" containerName="main" Jul 16 14:13:57.557713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557612 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:13:57.557713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557620 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:13:57.558194 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557719 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:13:57.558194 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.557734 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d10715e-0c52-494b-8881-acb9ea5e67f6" containerName="main" Jul 16 14:13:57.562757 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.562733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.565582 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.565562 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1305fb-openshift-default-dockercfg-qj6m6\"" Jul 16 14:13:57.583296 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.583271 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:13:57.652925 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.652864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.652929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.652972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbf5\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653095 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653282 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653282 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653282 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653206 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.653282 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.653253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.708501 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.708467 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:57.753763 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753728 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753928 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.753982 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.753972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754327 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbf5\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754327 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754327 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754485 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754485 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754371 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754485 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.754901 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.754860 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.756671 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.756646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.756956 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.756936 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.778810 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.778780 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.778978 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.778960 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbf5\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5\") pod \"e2e-api-a-1305fb-openshift-default-544994496f-ftkjt\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.873761 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.873656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:57.984616 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:57.984567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:58.027583 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.027559 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:13:58.029473 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:13:58.029445 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb202eb8_2bdf_4583_a155_9c943d34203e.slice/crio-aa806c61d24bfab3e493da742f5261d739e31d5f7634debd176885f68e574356 WatchSource:0}: Error finding container aa806c61d24bfab3e493da742f5261d739e31d5f7634debd176885f68e574356: Status 404 returned error can't find the container with id aa806c61d24bfab3e493da742f5261d739e31d5f7634debd176885f68e574356 Jul 16 14:13:58.031574 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.031543 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:13:58.031642 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.031610 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:13:58.031642 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.031636 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:13:58.804109 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.804071 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerStarted","Data":"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a"} Jul 16 14:13:58.804513 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.804126 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerStarted","Data":"aa806c61d24bfab3e493da742f5261d739e31d5f7634debd176885f68e574356"} Jul 16 14:13:58.829676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.829622 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podStartSLOduration=1.829605258 podStartE2EDuration="1.829605258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:13:58.826462048 +0000 UTC m=+2018.774578755" watchObservedRunningTime="2026-07-16 14:13:58.829605258 +0000 UTC m=+2018.777721965" Jul 16 14:13:58.874607 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:58.874575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:59.826788 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:59.826753 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:13:59.880569 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:59.880540 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:13:59.927574 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:13:59.927527 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:00.005552 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:00.005514 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:00.645579 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:00.645537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:00.745411 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:00.745376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:00.815959 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:00.815928 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:00.817015 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:00.816991 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:05.359516 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:05.359481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:05.928550 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:05.928513 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:06.251537 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:06.251429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:06.413366 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:06.413333 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:14:06.506336 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:06.506241 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:08.422417 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:08.422371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:08.538090 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:08.538054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:08.746543 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:08.746440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:08.968707 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:08.968603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:09.301065 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:09.301019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:09.403141 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:09.403094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:13.986546 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:13.986508 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:14.908742 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.908710 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerID="b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a" exitCode=0 Jul 16 14:14:14.908965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.908779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerDied","Data":"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a"} Jul 16 14:14:14.909218 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.909202 2563 scope.go:117] "RemoveContainer" containerID="b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a" Jul 16 14:14:14.910274 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.910247 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:14.910329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.910319 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:14.910364 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:14.910353 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:15.816076 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:15.816032 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:15.916702 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:15.916665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerStarted","Data":"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689"} Jul 16 14:14:16.874762 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:16.874716 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:16.875974 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:16.875948 2563 patch_prober.go:28] interesting pod/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 16 14:14:16.876085 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:16.876011 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 16 14:14:17.874516 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:17.874474 2563 patch_prober.go:28] interesting pod/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 16 14:14:17.874715 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:17.874543 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 16 14:14:18.875073 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:18.875032 2563 patch_prober.go:28] interesting pod/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 16 14:14:18.875592 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:18.875101 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 16 14:14:19.874831 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:19.874791 2563 patch_prober.go:28] interesting pod/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 16 14:14:19.875148 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:19.874866 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 16 14:14:20.878428 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:20.878397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:20.878800 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:20.878751 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:20.879521 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:20.879500 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:22.478444 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:22.478405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:22.577152 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:22.577105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:22.760426 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:22.760328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:24.719471 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:24.719432 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:14:28.821121 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:28.821085 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:31.335984 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:31.335946 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:31.462628 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:31.462597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:34.673082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:34.673039 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:34.752063 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:34.752021 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:34.982983 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:34.982893 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:14:34.983184 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:34.983161 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" containerID="cri-o://6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689" gracePeriod=30 Jul 16 14:14:35.001122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:35.001086 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:35.747705 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:35.747666 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:35.852112 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:35.852070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:35.881566 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:35.881479 2563 patch_prober.go:28] interesting pod/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 14:14:35.881769 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:35.881569 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 14:14:40.253727 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.253652 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:40.296136 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296106 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296152 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296178 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296241 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296276 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbf5\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296305 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296325 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296633 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296379 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296633 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo\") pod \"fb202eb8-2bdf-4583-a155-9c943d34203e\" (UID: \"fb202eb8-2bdf-4583-a155-9c943d34203e\") " Jul 16 14:14:40.296633 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296470 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:14:40.296796 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296657 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:14:40.296929 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296906 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.297025 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.296934 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-credential-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.297087 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.297032 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:14:40.297147 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.297103 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:14:40.297147 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.297115 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 14:14:40.298725 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.298696 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 14:14:40.299029 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.299002 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5" (OuterVolumeSpecName: "kube-api-access-6cbf5") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "kube-api-access-6cbf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:14:40.299029 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.299017 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:14:40.299164 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.299003 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb202eb8-2bdf-4583-a155-9c943d34203e" (UID: "fb202eb8-2bdf-4583-a155-9c943d34203e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:14:40.398014 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.397987 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398014 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398012 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbf5\" (UniqueName: \"kubernetes.io/projected/fb202eb8-2bdf-4583-a155-9c943d34203e-kube-api-access-6cbf5\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398023 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-workload-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398032 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb202eb8-2bdf-4583-a155-9c943d34203e-istiod-ca-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398040 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-podinfo\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398047 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-data\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:40.398220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:40.398055 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb202eb8-2bdf-4583-a155-9c943d34203e-istio-envoy\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:14:41.067405 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.067369 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerID="6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689" exitCode=0 Jul 16 14:14:41.067584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.067422 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" Jul 16 14:14:41.067584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.067452 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerDied","Data":"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689"} Jul 16 14:14:41.067584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.067511 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt" event={"ID":"fb202eb8-2bdf-4583-a155-9c943d34203e","Type":"ContainerDied","Data":"aa806c61d24bfab3e493da742f5261d739e31d5f7634debd176885f68e574356"} Jul 16 14:14:41.067584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.067534 2563 scope.go:117] "RemoveContainer" containerID="6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689" Jul 16 14:14:41.078854 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.078554 2563 scope.go:117] "RemoveContainer" containerID="b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a" Jul 16 14:14:41.086153 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.086129 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:14:41.089532 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.089512 2563 scope.go:117] "RemoveContainer" containerID="6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689" Jul 16 14:14:41.089790 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:14:41.089766 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689\": container with ID starting with 6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689 not found: ID does not exist" containerID="6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689" Jul 16 14:14:41.089915 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.089798 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689"} err="failed to get container status \"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689\": rpc error: code = NotFound desc = could not find container \"6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689\": container with ID starting with 6754db5aada7d4c72d0a994fefbe87825926318edebde5d61b9a85f8f34f0689 not found: ID does not exist" Jul 16 14:14:41.089915 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.089825 2563 scope.go:117] "RemoveContainer" containerID="b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a" Jul 16 14:14:41.090046 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.090003 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1305fb-openshift-default-544994496f-ftkjt"] Jul 16 14:14:41.090114 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:14:41.090095 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a\": container with ID starting with b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a not found: ID does not exist" containerID="b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a" Jul 16 14:14:41.090160 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.090120 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a"} err="failed to get container status \"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a\": rpc error: code = NotFound desc = could not find container \"b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a\": container with ID starting with b10c87cd5915c99544fc841a2188b82a5eb2c323f2b0c86604f3a9deebda654a not found: ID does not exist" Jul 16 14:14:41.685704 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:41.685661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:42.110604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:42.110564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:14:42.475349 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:42.475272 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" path="/var/lib/kubelet/pods/fb202eb8-2bdf-4583-a155-9c943d34203e/volumes" Jul 16 14:14:47.468994 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:47.468959 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:47.578927 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:47.578866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:47.790649 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:47.790574 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:48.087420 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.087337 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:48.449103 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.448968 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:48.765570 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.765540 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:14:48.766036 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766020 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.766094 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766038 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.766094 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766048 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.766094 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766054 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.766186 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766142 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.766186 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766151 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f672d71-f00d-47a9-8158-8c166c5cf1e7" containerName="istio-proxy" Jul 16 14:14:48.766186 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.766157 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb202eb8-2bdf-4583-a155-9c943d34203e" containerName="istio-proxy" Jul 16 14:14:48.770649 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.770632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.773172 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.773148 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c8253c-openshift-default-dockercfg-nbwfg\"" Jul 16 14:14:48.782507 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.782487 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:14:48.853836 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.853796 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:48.879105 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879290 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879290 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879212 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ljh\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879290 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879290 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879267 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879520 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879394 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879520 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879520 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.879520 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.879507 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980645 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980645 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980659 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980804 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ljh\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.980946 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.980929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.981350 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.981107 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.981350 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.981170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.981429 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.981364 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.981475 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.981421 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.981620 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.981603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.983577 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.983556 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.983733 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.983714 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.990465 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.990434 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:48.990465 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:48.990448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ljh\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh\") pod \"e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:49.081659 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.081570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:49.083414 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.083388 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:49.232264 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.232235 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:14:49.237522 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:14:49.237489 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb339ab84_e9ae_4c14_a8ac_9c850383b388.slice/crio-24bda3c53047f2c93286a01c628797d17132c4cfecd9a6ff2c17c4071117ba1f WatchSource:0}: Error finding container 24bda3c53047f2c93286a01c628797d17132c4cfecd9a6ff2c17c4071117ba1f: Status 404 returned error can't find the container with id 24bda3c53047f2c93286a01c628797d17132c4cfecd9a6ff2c17c4071117ba1f Jul 16 14:14:49.239428 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.239393 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:49.239547 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.239482 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:49.239547 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:49.239527 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:14:50.122695 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:50.122660 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerStarted","Data":"ba60f21399135ed39b18ccf2fe88a2080a150c28d7e274d53474bb554b2e1e28"} Jul 16 14:14:50.123224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:50.122720 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerStarted","Data":"24bda3c53047f2c93286a01c628797d17132c4cfecd9a6ff2c17c4071117ba1f"} Jul 16 14:14:50.145213 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:50.145153 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podStartSLOduration=2.145131036 podStartE2EDuration="2.145131036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:14:50.144055597 +0000 UTC m=+2070.092172327" watchObservedRunningTime="2026-07-16 14:14:50.145131036 +0000 UTC m=+2070.093247745" Jul 16 14:14:51.081791 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.081763 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:51.088495 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.088463 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:51.129193 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.129163 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:51.130217 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.130196 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:14:51.152713 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.152663 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:51.319172 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.319139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:51.417703 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.417601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:51.520398 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.520360 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:51.848627 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.848590 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:51.948920 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:51.948862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:52.805835 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:52.805796 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:14:56.711098 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:56.711063 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:57.084942 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:57.084910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:57.440451 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:57.440357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:58.042072 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:58.042028 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:58.207536 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:58.207498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:58.288007 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:58.287978 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:59.682943 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:59.682849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:59.768244 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:59.768211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:59.869729 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:59.869694 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:14:59.965913 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:14:59.965779 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:00.146954 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.146905 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn"] Jul 16 14:15:00.151397 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.151367 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8"] Jul 16 14:15:00.151553 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.151474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" Jul 16 14:15:00.155458 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.155432 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5bk8\"" Jul 16 14:15:00.156184 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.156169 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22"] Jul 16 14:15:00.156336 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.156311 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" Jul 16 14:15:00.160322 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.160292 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j"] Jul 16 14:15:00.160476 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.160372 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" Jul 16 14:15:00.166906 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.164492 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm"] Jul 16 14:15:00.166906 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.166720 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" Jul 16 14:15:00.167582 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.167427 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" Jul 16 14:15:00.168863 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.168832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn"] Jul 16 14:15:00.169005 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.168902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8"] Jul 16 14:15:00.169005 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.168995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" Jul 16 14:15:00.174150 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.174127 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22"] Jul 16 14:15:00.178148 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.178110 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm"] Jul 16 14:15:00.195397 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.195369 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j"] Jul 16 14:15:00.200222 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.200192 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj"] Jul 16 14:15:00.204409 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.204278 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg"] Jul 16 14:15:00.204575 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.204548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" Jul 16 14:15:00.207744 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.207722 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" Jul 16 14:15:00.207923 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.207838 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" Jul 16 14:15:00.208382 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.208356 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2"] Jul 16 14:15:00.213848 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.213825 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd"] Jul 16 14:15:00.216937 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.216760 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" Jul 16 14:15:00.231646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.227058 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv"] Jul 16 14:15:00.231646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.231466 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" Jul 16 14:15:00.232223 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.232048 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" Jul 16 14:15:00.232932 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.232558 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg"] Jul 16 14:15:00.232932 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.232657 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" Jul 16 14:15:00.233122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.233031 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2"] Jul 16 14:15:00.236376 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.236355 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj"] Jul 16 14:15:00.242091 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.241926 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv"] Jul 16 14:15:00.274368 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.268404 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t"] Jul 16 14:15:00.281285 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.281098 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" Jul 16 14:15:00.284151 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.284117 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd"] Jul 16 14:15:00.288213 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.287952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" Jul 16 14:15:00.297066 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.297036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t"] Jul 16 14:15:00.303188 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.302971 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" Jul 16 14:15:00.324866 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.324593 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" Jul 16 14:15:00.349429 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.347683 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" Jul 16 14:15:00.385417 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.385160 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" Jul 16 14:15:00.400000 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.399765 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" Jul 16 14:15:00.442903 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.442802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" Jul 16 14:15:00.460122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.460092 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" Jul 16 14:15:00.567543 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.567481 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8"] Jul 16 14:15:00.579716 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.579678 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937765e1_5368_4c8c_898f_1335c79c192b.slice/crio-b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5 WatchSource:0}: Error finding container b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5: Status 404 returned error can't find the container with id b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5 Jul 16 14:15:00.584040 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.583632 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 14:15:00.640229 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.639552 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn"] Jul 16 14:15:00.642062 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.641621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:00.666272 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.666220 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c434d0_a416_4e1a_a8c1_7a41d2db086f.slice/crio-9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c WatchSource:0}: Error finding container 9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c: Status 404 returned error can't find the container with id 9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c Jul 16 14:15:00.752602 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.752562 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:00.754596 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.754564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg"] Jul 16 14:15:00.755906 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.755855 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743a2269_8aff_4cee_b363_9ff832c906af.slice/crio-858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1 WatchSource:0}: Error finding container 858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1: Status 404 returned error can't find the container with id 858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1 Jul 16 14:15:00.796614 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.796569 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22"] Jul 16 14:15:00.798669 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.798609 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5ad6b7_96f7_4d73_ac34_dd574c69fd8d.slice/crio-79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013 WatchSource:0}: Error finding container 79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013: Status 404 returned error can't find the container with id 79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013 Jul 16 14:15:00.804807 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.803057 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j"] Jul 16 14:15:00.805894 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.805403 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5333d179_87cd_43cd_853a_8b0f8af23967.slice/crio-a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781 WatchSource:0}: Error finding container a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781: Status 404 returned error can't find the container with id a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781 Jul 16 14:15:00.846210 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.846180 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv"] Jul 16 14:15:00.848281 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.848243 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db68e6e_cab9_4fd7_ac1f_e1b949dda4cd.slice/crio-a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21 WatchSource:0}: Error finding container a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21: Status 404 returned error can't find the container with id a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21 Jul 16 14:15:00.890579 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.890553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm"] Jul 16 14:15:00.891843 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.891816 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0114c98_c8b5_4011_a57d_471ae4c6af1f.slice/crio-efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194 WatchSource:0}: Error finding container efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194: Status 404 returned error can't find the container with id efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194 Jul 16 14:15:00.982898 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.982851 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee14a4f_87cb_4dd9_811e_acc612c05579.slice/crio-5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd WatchSource:0}: Error finding container 5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd: Status 404 returned error can't find the container with id 5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd Jul 16 14:15:00.983429 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.983390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd"] Jul 16 14:15:00.984015 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.983986 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc929ea50_377d_439b_9d05_13da74665e2f.slice/crio-e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df WatchSource:0}: Error finding container e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df: Status 404 returned error can't find the container with id e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df Jul 16 14:15:00.985083 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:00.985002 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcfd902_58b5_45a0_9269_a2f8c5cad5e5.slice/crio-06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593 WatchSource:0}: Error finding container 06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593: Status 404 returned error can't find the container with id 06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593 Jul 16 14:15:00.986911 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.986891 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj"] Jul 16 14:15:00.989597 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:00.989576 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2"] Jul 16 14:15:01.057759 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.057729 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t"] Jul 16 14:15:01.059661 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:01.059630 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e753e93_c9f5_4a41_a4e8_fbe79c1f3206.slice/crio-982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666 WatchSource:0}: Error finding container 982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666: Status 404 returned error can't find the container with id 982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666 Jul 16 14:15:01.193316 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.193282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" event={"ID":"937765e1-5368-4c8c-898f-1335c79c192b","Type":"ContainerStarted","Data":"b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5"} Jul 16 14:15:01.194598 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.194571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerStarted","Data":"a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21"} Jul 16 14:15:01.195858 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.195833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" event={"ID":"ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d","Type":"ContainerStarted","Data":"79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013"} Jul 16 14:15:01.197154 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.197129 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" event={"ID":"0ee14a4f-87cb-4dd9-811e-acc612c05579","Type":"ContainerStarted","Data":"5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd"} Jul 16 14:15:01.198220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.198201 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" event={"ID":"c929ea50-377d-439b-9d05-13da74665e2f","Type":"ContainerStarted","Data":"e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df"} Jul 16 14:15:01.199436 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.199413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" event={"ID":"a0114c98-c8b5-4011-a57d-471ae4c6af1f","Type":"ContainerStarted","Data":"efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194"} Jul 16 14:15:01.200910 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.200867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" event={"ID":"9e753e93-c9f5-4a41-a4e8-fbe79c1f3206","Type":"ContainerStarted","Data":"982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666"} Jul 16 14:15:01.202828 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.202807 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" event={"ID":"5333d179-87cd-43cd-853a-8b0f8af23967","Type":"ContainerStarted","Data":"a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781"} Jul 16 14:15:01.204140 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.204114 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" event={"ID":"28c434d0-a416-4e1a-a8c1-7a41d2db086f","Type":"ContainerStarted","Data":"9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c"} Jul 16 14:15:01.205366 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.205336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" event={"ID":"743a2269-8aff-4cee-b363-9ff832c906af","Type":"ContainerStarted","Data":"858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1"} Jul 16 14:15:01.206450 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:01.206431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" event={"ID":"4fcfd902-58b5-45a0-9269-a2f8c5cad5e5","Type":"ContainerStarted","Data":"06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593"} Jul 16 14:15:01.586664 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:15:01.586620 2563 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/c2/c2e50d184eed6dc4c810c7883bd3854d3a15078b82f830637490b9bd1fd9ef20?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260716%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260716T141501Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=41fadc71e92f892d7d8bd3fff2f1a2666e7babbd43b4898b4742b07ad7baa829®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=ubi9----ubi-minimal&akamai_signature=exp=1784212201~hmac=45b3f23063eb2e17a07303d41ad0b2ead1221c85551c4a302643ae99deb077da\": EOF; artifact err: provided artifact is a container image" image="registry.redhat.io/ubi9/ubi-minimal:9.7" Jul 16 14:15:01.586895 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:15:01.586842 2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container cleanup start failed in pod maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2_odh-ai-gateway-infra(c929ea50-377d-439b-9d05-13da74665e2f): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/c2/c2e50d184eed6dc4c810c7883bd3854d3a15078b82f830637490b9bd1fd9ef20?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260716%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260716T141501Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=41fadc71e92f892d7d8bd3fff2f1a2666e7babbd43b4898b4742b07ad7baa829®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=ubi9----ubi-minimal&akamai_signature=exp=1784212201~hmac=45b3f23063eb2e17a07303d41ad0b2ead1221c85551c4a302643ae99deb077da\": EOF; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 16 14:15:01.587002 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:15:01.586936 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/c2/c2e50d184eed6dc4c810c7883bd3854d3a15078b82f830637490b9bd1fd9ef20?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260716%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260716T141501Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=41fadc71e92f892d7d8bd3fff2f1a2666e7babbd43b4898b4742b07ad7baa829®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=ubi9----ubi-minimal&akamai_signature=exp=1784212201~hmac=45b3f23063eb2e17a07303d41ad0b2ead1221c85551c4a302643ae99deb077da\\\": EOF; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" podUID="c929ea50-377d-439b-9d05-13da74665e2f" Jul 16 14:15:02.222611 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:15:02.222543 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/c2/c2e50d184eed6dc4c810c7883bd3854d3a15078b82f830637490b9bd1fd9ef20?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260716%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260716T141501Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=41fadc71e92f892d7d8bd3fff2f1a2666e7babbd43b4898b4742b07ad7baa829®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=ubi9----ubi-minimal&akamai_signature=exp=1784212201~hmac=45b3f23063eb2e17a07303d41ad0b2ead1221c85551c4a302643ae99deb077da\\\": EOF; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" podUID="c929ea50-377d-439b-9d05-13da74665e2f" Jul 16 14:15:05.267280 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:05.267218 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:06.306100 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.306063 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" event={"ID":"a0114c98-c8b5-4011-a57d-471ae4c6af1f","Type":"ContainerStarted","Data":"4885819f869a8cc89cf53dcabed097c96ea26a215da36938d75f9aa196cd767a"} Jul 16 14:15:06.308271 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.308241 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" event={"ID":"9e753e93-c9f5-4a41-a4e8-fbe79c1f3206","Type":"ContainerStarted","Data":"5995c202ee180259ffab1536e527c635a71b12509d30f67cc2af699581d59635"} Jul 16 14:15:06.309846 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.309819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" event={"ID":"28c434d0-a416-4e1a-a8c1-7a41d2db086f","Type":"ContainerStarted","Data":"5787710a45ed8e2891b329d56e6965378111819fff7c6b7cb68a5688874d1d9a"} Jul 16 14:15:06.311400 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.311375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" event={"ID":"743a2269-8aff-4cee-b363-9ff832c906af","Type":"ContainerStarted","Data":"f5b6c599764d3b249b86f9143f9d1f300d092aa488f35acf84a7baec8e1596e7"} Jul 16 14:15:06.312894 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.312857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" event={"ID":"4fcfd902-58b5-45a0-9269-a2f8c5cad5e5","Type":"ContainerStarted","Data":"d511a6a5cf2419b0af1c5d6fc7aa3d2f1e178064e4acbb9bdcbe81057b970de0"} Jul 16 14:15:06.314385 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.314363 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerStarted","Data":"8bb631a53a5a1ef2377a6c334ffd35335169c605d70be208744c457d172791b3"} Jul 16 14:15:06.321395 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.321314 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" podStartSLOduration=1.326447348 podStartE2EDuration="6.321301309s" totalImagesPullingTime="4.994853961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.320721798 +0000 UTC m=+2086.268838504" watchObservedRunningTime="2026-07-16 14:15:06.321301309 +0000 UTC m=+2086.269418016" Jul 16 14:15:06.360072 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.359995 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" podStartSLOduration=1.6761460700000002 podStartE2EDuration="6.359970101s" totalImagesPullingTime="4.683824031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.347639422 +0000 UTC m=+2086.295756150" watchObservedRunningTime="2026-07-16 14:15:06.359970101 +0000 UTC m=+2086.308086809" Jul 16 14:15:06.376215 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.375267 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" podStartSLOduration=1.58868982 podStartE2EDuration="6.375242518s" totalImagesPullingTime="4.786552698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.370110835 +0000 UTC m=+2086.318227545" watchObservedRunningTime="2026-07-16 14:15:06.375242518 +0000 UTC m=+2086.323359224" Jul 16 14:15:06.392767 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.392718 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" podStartSLOduration=1.475166267 podStartE2EDuration="6.392700182s" totalImagesPullingTime="4.917533915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.38995801 +0000 UTC m=+2086.338074720" watchObservedRunningTime="2026-07-16 14:15:06.392700182 +0000 UTC m=+2086.340816889" Jul 16 14:15:06.408315 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.408196 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" podStartSLOduration=1.406687158 podStartE2EDuration="6.408178357s" totalImagesPullingTime="5.001491199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.406825042 +0000 UTC m=+2086.354941750" watchObservedRunningTime="2026-07-16 14:15:06.408178357 +0000 UTC m=+2086.356295055" Jul 16 14:15:06.424858 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:06.424798 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" podStartSLOduration=1.75183534 podStartE2EDuration="6.424783344s" totalImagesPullingTime="4.672948004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:06.422950004 +0000 UTC m=+2086.371066732" watchObservedRunningTime="2026-07-16 14:15:06.424783344 +0000 UTC m=+2086.372900052" Jul 16 14:15:07.323314 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.323283 2563 generic.go:356] "Generic (PLEG): container finished" podID="9e753e93-c9f5-4a41-a4e8-fbe79c1f3206" containerID="5995c202ee180259ffab1536e527c635a71b12509d30f67cc2af699581d59635" exitCode=0 Jul 16 14:15:07.323727 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.323366 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" event={"ID":"9e753e93-c9f5-4a41-a4e8-fbe79c1f3206","Type":"ContainerDied","Data":"5995c202ee180259ffab1536e527c635a71b12509d30f67cc2af699581d59635"} Jul 16 14:15:07.325047 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.325023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" event={"ID":"5333d179-87cd-43cd-853a-8b0f8af23967","Type":"ContainerStarted","Data":"6b8f1e15c0f499c928f9e10f1be224621299e7148ce424be8b375d526fa90b63"} Jul 16 14:15:07.326493 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.326471 2563 generic.go:356] "Generic (PLEG): container finished" podID="28c434d0-a416-4e1a-a8c1-7a41d2db086f" containerID="5787710a45ed8e2891b329d56e6965378111819fff7c6b7cb68a5688874d1d9a" exitCode=0 Jul 16 14:15:07.326593 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.326549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" event={"ID":"28c434d0-a416-4e1a-a8c1-7a41d2db086f","Type":"ContainerDied","Data":"5787710a45ed8e2891b329d56e6965378111819fff7c6b7cb68a5688874d1d9a"} Jul 16 14:15:07.328138 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.328116 2563 generic.go:356] "Generic (PLEG): container finished" podID="743a2269-8aff-4cee-b363-9ff832c906af" containerID="f5b6c599764d3b249b86f9143f9d1f300d092aa488f35acf84a7baec8e1596e7" exitCode=0 Jul 16 14:15:07.328239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.328182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" event={"ID":"743a2269-8aff-4cee-b363-9ff832c906af","Type":"ContainerDied","Data":"f5b6c599764d3b249b86f9143f9d1f300d092aa488f35acf84a7baec8e1596e7"} Jul 16 14:15:07.329554 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.329535 2563 generic.go:356] "Generic (PLEG): container finished" podID="4fcfd902-58b5-45a0-9269-a2f8c5cad5e5" containerID="d511a6a5cf2419b0af1c5d6fc7aa3d2f1e178064e4acbb9bdcbe81057b970de0" exitCode=0 Jul 16 14:15:07.329641 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.329562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" event={"ID":"4fcfd902-58b5-45a0-9269-a2f8c5cad5e5","Type":"ContainerDied","Data":"d511a6a5cf2419b0af1c5d6fc7aa3d2f1e178064e4acbb9bdcbe81057b970de0"} Jul 16 14:15:07.331039 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.331017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" event={"ID":"937765e1-5368-4c8c-898f-1335c79c192b","Type":"ContainerStarted","Data":"cfd37015d29d9b85d8a01499ffca4905251591dd9dd7cb4480ec4d60b1109fbe"} Jul 16 14:15:07.332370 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.332342 2563 generic.go:356] "Generic (PLEG): container finished" podID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerID="8bb631a53a5a1ef2377a6c334ffd35335169c605d70be208744c457d172791b3" exitCode=7 Jul 16 14:15:07.332460 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.332425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerDied","Data":"8bb631a53a5a1ef2377a6c334ffd35335169c605d70be208744c457d172791b3"} Jul 16 14:15:07.332533 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.332519 2563 scope.go:117] "RemoveContainer" containerID="8bb631a53a5a1ef2377a6c334ffd35335169c605d70be208744c457d172791b3" Jul 16 14:15:07.333851 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.333828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" event={"ID":"ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d","Type":"ContainerStarted","Data":"1b4b3ce97574173b0735a11952a82bf9e642c453a27c74e258cb8cfe6afc62b5"} Jul 16 14:15:07.335207 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.335179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" event={"ID":"0ee14a4f-87cb-4dd9-811e-acc612c05579","Type":"ContainerStarted","Data":"08b312459cd6ae00e771ca5ee745aba0bf886f899f207c64aa1760f3c74bc0e9"} Jul 16 14:15:07.336680 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.336661 2563 generic.go:356] "Generic (PLEG): container finished" podID="a0114c98-c8b5-4011-a57d-471ae4c6af1f" containerID="4885819f869a8cc89cf53dcabed097c96ea26a215da36938d75f9aa196cd767a" exitCode=0 Jul 16 14:15:07.336774 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.336700 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" event={"ID":"a0114c98-c8b5-4011-a57d-471ae4c6af1f","Type":"ContainerDied","Data":"4885819f869a8cc89cf53dcabed097c96ea26a215da36938d75f9aa196cd767a"} Jul 16 14:15:07.379277 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.379222 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" podStartSLOduration=1.646857875 podStartE2EDuration="7.379205686s" totalImagesPullingTime="5.732347811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:07.378026326 +0000 UTC m=+2087.326143034" watchObservedRunningTime="2026-07-16 14:15:07.379205686 +0000 UTC m=+2087.327322394" Jul 16 14:15:07.413836 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.413734 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" podStartSLOduration=2.005771821 podStartE2EDuration="7.413718871s" totalImagesPullingTime="5.40794705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:07.410295494 +0000 UTC m=+2087.358412201" watchObservedRunningTime="2026-07-16 14:15:07.413718871 +0000 UTC m=+2087.361835577" Jul 16 14:15:07.446514 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.446452 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" podStartSLOduration=2.335189424 podStartE2EDuration="7.446430346s" totalImagesPullingTime="5.111240922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:07.442219844 +0000 UTC m=+2087.390336552" watchObservedRunningTime="2026-07-16 14:15:07.446430346 +0000 UTC m=+2087.394547054" Jul 16 14:15:07.476450 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:07.476399 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" podStartSLOduration=2.290605525 podStartE2EDuration="7.476382498s" totalImagesPullingTime="5.185776973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:07.47502788 +0000 UTC m=+2087.423144592" watchObservedRunningTime="2026-07-16 14:15:07.476382498 +0000 UTC m=+2087.424499201" Jul 16 14:15:08.348363 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.348322 2563 generic.go:356] "Generic (PLEG): container finished" podID="5333d179-87cd-43cd-853a-8b0f8af23967" containerID="6b8f1e15c0f499c928f9e10f1be224621299e7148ce424be8b375d526fa90b63" exitCode=0 Jul 16 14:15:08.348811 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.348415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" event={"ID":"5333d179-87cd-43cd-853a-8b0f8af23967","Type":"ContainerDied","Data":"6b8f1e15c0f499c928f9e10f1be224621299e7148ce424be8b375d526fa90b63"} Jul 16 14:15:08.350289 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.350263 2563 generic.go:356] "Generic (PLEG): container finished" podID="937765e1-5368-4c8c-898f-1335c79c192b" containerID="cfd37015d29d9b85d8a01499ffca4905251591dd9dd7cb4480ec4d60b1109fbe" exitCode=0 Jul 16 14:15:08.350402 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.350345 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" event={"ID":"937765e1-5368-4c8c-898f-1335c79c192b","Type":"ContainerDied","Data":"cfd37015d29d9b85d8a01499ffca4905251591dd9dd7cb4480ec4d60b1109fbe"} Jul 16 14:15:08.352496 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.352477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerStarted","Data":"508aebed245bd0f9455c11f776b034a05730b292a86b8b72e859afd5167686e2"} Jul 16 14:15:08.354119 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.354094 2563 generic.go:356] "Generic (PLEG): container finished" podID="ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d" containerID="1b4b3ce97574173b0735a11952a82bf9e642c453a27c74e258cb8cfe6afc62b5" exitCode=0 Jul 16 14:15:08.354207 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.354185 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" event={"ID":"ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d","Type":"ContainerDied","Data":"1b4b3ce97574173b0735a11952a82bf9e642c453a27c74e258cb8cfe6afc62b5"} Jul 16 14:15:08.355707 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.355684 2563 generic.go:356] "Generic (PLEG): container finished" podID="0ee14a4f-87cb-4dd9-811e-acc612c05579" containerID="08b312459cd6ae00e771ca5ee745aba0bf886f899f207c64aa1760f3c74bc0e9" exitCode=0 Jul 16 14:15:08.355803 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.355781 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" event={"ID":"0ee14a4f-87cb-4dd9-811e-acc612c05579","Type":"ContainerDied","Data":"08b312459cd6ae00e771ca5ee745aba0bf886f899f207c64aa1760f3c74bc0e9"} Jul 16 14:15:08.564483 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.564456 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" Jul 16 14:15:08.572023 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.571987 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" Jul 16 14:15:08.724289 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.724265 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" Jul 16 14:15:08.727843 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.727819 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" Jul 16 14:15:08.731655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:08.731634 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" Jul 16 14:15:09.366093 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.366050 2563 generic.go:356] "Generic (PLEG): container finished" podID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerID="508aebed245bd0f9455c11f776b034a05730b292a86b8b72e859afd5167686e2" exitCode=0 Jul 16 14:15:09.366557 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.366143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerDied","Data":"508aebed245bd0f9455c11f776b034a05730b292a86b8b72e859afd5167686e2"} Jul 16 14:15:09.366557 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.366243 2563 scope.go:117] "RemoveContainer" containerID="8bb631a53a5a1ef2377a6c334ffd35335169c605d70be208744c457d172791b3" Jul 16 14:15:09.367834 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.367809 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" event={"ID":"a0114c98-c8b5-4011-a57d-471ae4c6af1f","Type":"ContainerDied","Data":"efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194"} Jul 16 14:15:09.367834 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.367821 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm" Jul 16 14:15:09.368018 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.367854 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc2d8ed9353c69f4b22d26c44a3d6d18b2b63189796c71cdb0b8f8212bb3194" Jul 16 14:15:09.369369 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.369285 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" Jul 16 14:15:09.369369 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.369317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t" event={"ID":"9e753e93-c9f5-4a41-a4e8-fbe79c1f3206","Type":"ContainerDied","Data":"982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666"} Jul 16 14:15:09.369369 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.369357 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982d91692bfcdb07ca18344a1efab3b210e95ddfc59cc75653aac43b368ee666" Jul 16 14:15:09.371497 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.371474 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" Jul 16 14:15:09.371497 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.371483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn" event={"ID":"28c434d0-a416-4e1a-a8c1-7a41d2db086f","Type":"ContainerDied","Data":"9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c"} Jul 16 14:15:09.371652 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.371522 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7103211704b7259e0fabb645442f60c98b01eb25ffac4e2db4e5530deda62c" Jul 16 14:15:09.373125 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.373074 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" event={"ID":"743a2269-8aff-4cee-b363-9ff832c906af","Type":"ContainerDied","Data":"858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1"} Jul 16 14:15:09.373225 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.373146 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858c5c7d9319c53a717e0e37b20b9d84066abf73c088b171e84126ecdaca7ed1" Jul 16 14:15:09.373225 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.373207 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg" Jul 16 14:15:09.376361 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.376286 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" Jul 16 14:15:09.376361 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.376274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj" event={"ID":"4fcfd902-58b5-45a0-9269-a2f8c5cad5e5","Type":"ContainerDied","Data":"06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593"} Jul 16 14:15:09.376361 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.376333 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ca8112cb122c92160891112718e4ecf057bc78baa87dd8dc5c494ff4514593" Jul 16 14:15:09.563272 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.562818 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" Jul 16 14:15:09.614767 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.614336 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" Jul 16 14:15:09.618711 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.618589 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" Jul 16 14:15:09.622939 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:09.622918 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" Jul 16 14:15:10.385321 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.385288 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" Jul 16 14:15:10.385321 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.385296 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736855-w4fd8" event={"ID":"937765e1-5368-4c8c-898f-1335c79c192b","Type":"ContainerDied","Data":"b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5"} Jul 16 14:15:10.385864 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.385343 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f57447a3a6442674ca3821c90f31e8ca6ac343b4b43a58deefe0818b7233b5" Jul 16 14:15:10.389205 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.389159 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" Jul 16 14:15:10.389205 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.389175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22" event={"ID":"ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d","Type":"ContainerDied","Data":"79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013"} Jul 16 14:15:10.389387 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.389231 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c6c8899c7cb6c0d320177a00bf4d4d291c855dac37952f8a9d918d3edf8013" Jul 16 14:15:10.391849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.390939 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" Jul 16 14:15:10.391849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.391034 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd" event={"ID":"0ee14a4f-87cb-4dd9-811e-acc612c05579","Type":"ContainerDied","Data":"5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd"} Jul 16 14:15:10.391849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.391075 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c18e044881291716e50c56c002dc3508042bec4bc937f090411309b58c0ccfd" Jul 16 14:15:10.393288 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.392608 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" event={"ID":"5333d179-87cd-43cd-853a-8b0f8af23967","Type":"ContainerDied","Data":"a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781"} Jul 16 14:15:10.393288 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.392654 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d3700f2416fdf7f2da3b62782b809f6ec9352269a8af562243ae09a9ec6781" Jul 16 14:15:10.393288 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.392710 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j" Jul 16 14:15:10.551486 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:10.551456 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" Jul 16 14:15:11.400475 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.400452 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" Jul 16 14:15:11.400475 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.400462 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv" event={"ID":"4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd","Type":"ContainerDied","Data":"a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21"} Jul 16 14:15:11.400986 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.400546 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60558f5a6adc8fe14014fc3d12580230db92358758a240ff164082ec7c14d21" Jul 16 14:15:11.402221 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.402193 2563 generic.go:356] "Generic (PLEG): container finished" podID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerID="ba60f21399135ed39b18ccf2fe88a2080a150c28d7e274d53474bb554b2e1e28" exitCode=0 Jul 16 14:15:11.402324 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.402264 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerDied","Data":"ba60f21399135ed39b18ccf2fe88a2080a150c28d7e274d53474bb554b2e1e28"} Jul 16 14:15:11.402664 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.402651 2563 scope.go:117] "RemoveContainer" containerID="ba60f21399135ed39b18ccf2fe88a2080a150c28d7e274d53474bb554b2e1e28" Jul 16 14:15:11.403819 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.403790 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:15:11.403923 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.403907 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:15:11.403969 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:11.403949 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 14:15:12.411658 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:12.411622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerStarted","Data":"0744715aae1b5458883afc598b71ee031a8314bc3b91efffbf10786e854831f9"} Jul 16 14:15:12.611940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:12.611905 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:15:13.082114 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:13.082074 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:15:13.082425 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:13.082399 2563 patch_prober.go:28] interesting pod/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 16 14:15:13.082548 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:13.082448 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 16 14:15:13.607607 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:13.607570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:13.708453 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:13.708421 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:14.082658 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:14.082618 2563 patch_prober.go:28] interesting pod/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 16 14:15:14.082847 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:14.082679 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 16 14:15:15.082703 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:15.082663 2563 patch_prober.go:28] interesting pod/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 16 14:15:15.083141 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:15.082721 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 16 14:15:16.082196 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:16.082158 2563 patch_prober.go:28] interesting pod/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 16 14:15:16.082391 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:16.082221 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 16 14:15:17.086438 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.086406 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:15:17.086825 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.086691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:15:17.087366 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.087348 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:15:17.451224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.451133 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" event={"ID":"c929ea50-377d-439b-9d05-13da74665e2f","Type":"ContainerStarted","Data":"00e12560250241a98ebcc090816f6c2477dec92fe28e04ad75f2d6f42a9547b4"} Jul 16 14:15:17.470738 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.470678 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" podStartSLOduration=17.470661924 podStartE2EDuration="17.470661924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-16 14:15:17.467954449 +0000 UTC m=+2097.416071174" watchObservedRunningTime="2026-07-16 14:15:17.470661924 +0000 UTC m=+2097.418778631" Jul 16 14:15:17.852063 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852030 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:15:17.852555 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852537 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5333d179-87cd-43cd-853a-8b0f8af23967" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852558 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5333d179-87cd-43cd-853a-8b0f8af23967" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852572 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcfd902-58b5-45a0-9269-a2f8c5cad5e5" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852579 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcfd902-58b5-45a0-9269-a2f8c5cad5e5" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852587 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c434d0-a416-4e1a-a8c1-7a41d2db086f" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852596 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c434d0-a416-4e1a-a8c1-7a41d2db086f" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852606 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e753e93-c9f5-4a41-a4e8-fbe79c1f3206" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852614 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e753e93-c9f5-4a41-a4e8-fbe79c1f3206" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852630 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937765e1-5368-4c8c-898f-1335c79c192b" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852638 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="937765e1-5368-4c8c-898f-1335c79c192b" containerName="cleanup" Jul 16 14:15:17.852654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852656 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee14a4f-87cb-4dd9-811e-acc612c05579" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852664 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee14a4f-87cb-4dd9-811e-acc612c05579" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852674 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852682 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852692 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852702 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852714 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743a2269-8aff-4cee-b363-9ff832c906af" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852722 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="743a2269-8aff-4cee-b363-9ff832c906af" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852748 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852758 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852769 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0114c98-c8b5-4011-a57d-471ae4c6af1f" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852778 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0114c98-c8b5-4011-a57d-471ae4c6af1f" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852905 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee14a4f-87cb-4dd9-811e-acc612c05579" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852919 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852932 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e753e93-c9f5-4a41-a4e8-fbe79c1f3206" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852943 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="743a2269-8aff-4cee-b363-9ff832c906af" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852952 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcfd902-58b5-45a0-9269-a2f8c5cad5e5" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852961 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c434d0-a416-4e1a-a8c1-7a41d2db086f" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852972 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5333d179-87cd-43cd-853a-8b0f8af23967" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852983 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.852991 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="937765e1-5368-4c8c-898f-1335c79c192b" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.853005 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0114c98-c8b5-4011-a57d-471ae4c6af1f" containerName="cleanup" Jul 16 14:15:17.853200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.853013 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd" containerName="cleanup" Jul 16 14:15:17.856704 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.856683 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.859554 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.859533 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8253c\"/\"kube-root-ca.crt\"" Jul 16 14:15:17.860115 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.860097 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8253c\"/\"openshift-service-ca.crt\"" Jul 16 14:15:17.860626 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.860607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8253c\"/\"default-dockercfg-8mfm5\"" Jul 16 14:15:17.860722 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.860631 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8253c\"/\"auth-test-model-c8253c-kserve-self-signed-certs\"" Jul 16 14:15:17.868975 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.868951 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:15:17.898390 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztplp\" (UniqueName: \"kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.898556 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.898556 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.898556 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898507 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.898556 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.898708 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.898583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.908676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.908645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:17.999432 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999646 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztplp\" (UniqueName: \"kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999910 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999971 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999864 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:17.999971 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999932 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.000047 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:17.999990 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.001860 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.001842 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.002053 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.002036 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.010337 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.010306 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztplp\" (UniqueName: \"kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp\") pod \"auth-test-model-c8253c-kserve-7899dbfc87-qfk8j\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.166561 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.166457 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:18.320560 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.320533 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:15:18.321582 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:15:18.321560 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e53cc20_ab31_481f_9ecd_d26949d865d9.slice/crio-c0e92a144c0b1df498a27cda58d590fb8deb3329428eca3c03fb450fa0429782 WatchSource:0}: Error finding container c0e92a144c0b1df498a27cda58d590fb8deb3329428eca3c03fb450fa0429782: Status 404 returned error can't find the container with id c0e92a144c0b1df498a27cda58d590fb8deb3329428eca3c03fb450fa0429782 Jul 16 14:15:18.460419 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.460333 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" event={"ID":"4e53cc20-ab31-481f-9ecd-d26949d865d9","Type":"ContainerStarted","Data":"864684075e6d4b8f80a00cf3c377599c15f748b52362b830b77dc8651753c14a"} Jul 16 14:15:18.460419 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.460380 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" event={"ID":"4e53cc20-ab31-481f-9ecd-d26949d865d9","Type":"ContainerStarted","Data":"c0e92a144c0b1df498a27cda58d590fb8deb3329428eca3c03fb450fa0429782"} Jul 16 14:15:18.461904 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.461864 2563 generic.go:356] "Generic (PLEG): container finished" podID="c929ea50-377d-439b-9d05-13da74665e2f" containerID="00e12560250241a98ebcc090816f6c2477dec92fe28e04ad75f2d6f42a9547b4" exitCode=0 Jul 16 14:15:18.462015 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.461937 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" event={"ID":"c929ea50-377d-439b-9d05-13da74665e2f","Type":"ContainerDied","Data":"00e12560250241a98ebcc090816f6c2477dec92fe28e04ad75f2d6f42a9547b4"} Jul 16 14:15:18.485004 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:18.484960 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" podStartSLOduration=1.484944333 podStartE2EDuration="1.484944333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:15:18.48213805 +0000 UTC m=+2098.430254757" watchObservedRunningTime="2026-07-16 14:15:18.484944333 +0000 UTC m=+2098.433061039" Jul 16 14:15:19.609274 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:19.609246 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" Jul 16 14:15:20.476733 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:20.476702 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" Jul 16 14:15:20.476956 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:20.476910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2" event={"ID":"c929ea50-377d-439b-9d05-13da74665e2f","Type":"ContainerDied","Data":"e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df"} Jul 16 14:15:20.477020 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:20.476956 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20ef933001f963c184bae128fc9651675c5797840b380d7c5b7171fe563d8df" Jul 16 14:15:20.852228 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:20.852202 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:15:20.860221 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:20.860194 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:15:21.579406 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:21.579369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:21.679929 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:21.679867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:24.822723 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:24.822691 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:15:28.167178 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:28.167149 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:28.167178 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:28.167186 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:28.180205 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:28.180170 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:28.537773 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:28.537742 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:15:42.074527 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:42.074493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:43.296280 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:43.296246 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:43.374166 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:43.374137 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:46.004890 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:46.004843 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:15:51.435991 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:51.435952 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:15:51.716111 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:15:51.716027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:16:02.307707 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:16:02.307668 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:16:22.822579 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:16:22.822539 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:16:30.508150 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:16:30.508096 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:16:42.799551 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:16:42.799514 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:17:04.108294 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:17:04.108255 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:17:24.931553 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:17:24.931513 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:17:44.501209 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:17:44.501129 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:18:01.382668 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:01.382632 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:18:01.480655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:01.480599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:18:02.325319 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:02.325280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:18:05.803576 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:05.803539 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:18:06.463915 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:06.463867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:18:16.403596 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.403537 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" podUID="4e53cc20-ab31-481f-9ecd-d26949d865d9" containerName="main" containerID="cri-o://864684075e6d4b8f80a00cf3c377599c15f748b52362b830b77dc8651753c14a" gracePeriod=45 Jul 16 14:18:16.631731 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.631699 2563 generic.go:356] "Generic (PLEG): container finished" podID="4e53cc20-ab31-481f-9ecd-d26949d865d9" containerID="864684075e6d4b8f80a00cf3c377599c15f748b52362b830b77dc8651753c14a" exitCode=0 Jul 16 14:18:16.631934 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.631769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" event={"ID":"4e53cc20-ab31-481f-9ecd-d26949d865d9","Type":"ContainerDied","Data":"864684075e6d4b8f80a00cf3c377599c15f748b52362b830b77dc8651753c14a"} Jul 16 14:18:16.667256 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.667191 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:18:16.807230 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807186 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807263 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztplp\" (UniqueName: \"kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807303 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807327 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807359 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807400 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache\") pod \"4e53cc20-ab31-481f-9ecd-d26949d865d9\" (UID: \"4e53cc20-ab31-481f-9ecd-d26949d865d9\") " Jul 16 14:18:16.807540 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807497 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home" (OuterVolumeSpecName: "home") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:18:16.807825 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807664 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:18:16.807825 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807710 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:18:16.807951 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807847 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-home\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:16.807951 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807896 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-tmp-dir\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:16.807951 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.807919 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-model-cache\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:16.809655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.809623 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm" (OuterVolumeSpecName: "dshm") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:18:16.809655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.809644 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp" (OuterVolumeSpecName: "kube-api-access-ztplp") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "kube-api-access-ztplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:18:16.809827 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.809660 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e53cc20-ab31-481f-9ecd-d26949d865d9" (UID: "4e53cc20-ab31-481f-9ecd-d26949d865d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 14:18:16.909119 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.909079 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztplp\" (UniqueName: \"kubernetes.io/projected/4e53cc20-ab31-481f-9ecd-d26949d865d9-kube-api-access-ztplp\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:16.909119 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.909112 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e53cc20-ab31-481f-9ecd-d26949d865d9-tls-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:16.909119 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:16.909123 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e53cc20-ab31-481f-9ecd-d26949d865d9-dshm\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:18:17.640223 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:17.640192 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" Jul 16 14:18:17.640716 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:17.640188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j" event={"ID":"4e53cc20-ab31-481f-9ecd-d26949d865d9","Type":"ContainerDied","Data":"c0e92a144c0b1df498a27cda58d590fb8deb3329428eca3c03fb450fa0429782"} Jul 16 14:18:17.640716 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:17.640328 2563 scope.go:117] "RemoveContainer" containerID="864684075e6d4b8f80a00cf3c377599c15f748b52362b830b77dc8651753c14a" Jul 16 14:18:17.664373 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:17.664341 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:18:17.668160 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:17.668135 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c8253c/auth-test-model-c8253c-kserve-7899dbfc87-qfk8j"] Jul 16 14:18:18.475014 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:18.474971 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e53cc20-ab31-481f-9ecd-d26949d865d9" path="/var/lib/kubelet/pods/4e53cc20-ab31-481f-9ecd-d26949d865d9/volumes" Jul 16 14:18:25.308448 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:25.308403 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:18:35.735628 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.735597 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:18:35.736089 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736074 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e53cc20-ab31-481f-9ecd-d26949d865d9" containerName="main" Jul 16 14:18:35.736089 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736090 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e53cc20-ab31-481f-9ecd-d26949d865d9" containerName="main" Jul 16 14:18:35.736166 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736099 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c929ea50-377d-439b-9d05-13da74665e2f" containerName="cleanup" Jul 16 14:18:35.736166 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736105 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c929ea50-377d-439b-9d05-13da74665e2f" containerName="cleanup" Jul 16 14:18:35.736233 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736186 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e53cc20-ab31-481f-9ecd-d26949d865d9" containerName="main" Jul 16 14:18:35.736233 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.736201 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c929ea50-377d-439b-9d05-13da74665e2f" containerName="cleanup" Jul 16 14:18:35.739683 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.739665 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.743017 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.742992 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"sub-test-model-1f53bd-kserve-self-signed-certs\"" Jul 16 14:18:35.743204 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.743011 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"default-dockercfg-7pkpp\"" Jul 16 14:18:35.743204 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.743011 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"openshift-service-ca.crt\"" Jul 16 14:18:35.743397 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.743096 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"kube-root-ca.crt\"" Jul 16 14:18:35.749116 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.749091 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:18:35.767501 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.767465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:18:35.791364 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.791545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.791545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.791545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791470 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.791545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.791711 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.791565 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2tnp\" (UniqueName: \"kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893106 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893313 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893313 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893313 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893169 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893313 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tnp\" (UniqueName: \"kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893313 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893242 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893584 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893666 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.893710 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.893675 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.895465 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.895445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.895734 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.895716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:35.902075 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:35.902049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2tnp\" (UniqueName: \"kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp\") pod \"sub-test-model-1f53bd-kserve-5ff7895577-zmntf\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:36.052012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:36.051966 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:36.194964 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:36.194937 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:18:36.197226 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:18:36.197196 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fb35bfd_1724_49ce_892a_51c18d6f7eec.slice/crio-46817ba88a0e1a45fc95748341afa45228e45b6fcbed1b8b392856b66e84c9bd WatchSource:0}: Error finding container 46817ba88a0e1a45fc95748341afa45228e45b6fcbed1b8b392856b66e84c9bd: Status 404 returned error can't find the container with id 46817ba88a0e1a45fc95748341afa45228e45b6fcbed1b8b392856b66e84c9bd Jul 16 14:18:36.762751 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:36.762715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" event={"ID":"2fb35bfd-1724-49ce-892a-51c18d6f7eec","Type":"ContainerStarted","Data":"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528"} Jul 16 14:18:36.763169 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:36.762765 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" event={"ID":"2fb35bfd-1724-49ce-892a-51c18d6f7eec","Type":"ContainerStarted","Data":"46817ba88a0e1a45fc95748341afa45228e45b6fcbed1b8b392856b66e84c9bd"} Jul 16 14:18:36.779343 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:36.779288 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" podStartSLOduration=1.7792722410000001 podStartE2EDuration="1.779272241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:18:36.778157556 +0000 UTC m=+2296.726274267" watchObservedRunningTime="2026-07-16 14:18:36.779272241 +0000 UTC m=+2296.727388977" Jul 16 14:18:42.799493 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:42.799452 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:18:46.052286 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:46.052247 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:46.052690 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:46.052307 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:46.064868 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:46.064842 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:46.841518 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:46.841487 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:18:59.900707 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:18:59.900659 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:19:07.401452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:07.401414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:19:22.618533 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:22.618397 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:19:28.792200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:28.792158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:19:29.515092 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:29.515057 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:19:29.537482 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:29.537447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:19:36.074048 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:36.074008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:19:44.101277 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.101232 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:19:44.535560 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.535487 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" podUID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" containerName="main" containerID="cri-o://3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528" gracePeriod=45 Jul 16 14:19:44.802530 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.802466 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:19:44.872789 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.872752 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.872812 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.872837 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.872924 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2tnp\" (UniqueName: \"kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.872955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873231 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873021 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs\") pod \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\" (UID: \"2fb35bfd-1724-49ce-892a-51c18d6f7eec\") " Jul 16 14:19:44.873231 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873055 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:19:44.873231 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873090 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home" (OuterVolumeSpecName: "home") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:19:44.873374 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873307 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:19:44.873374 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873327 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-model-cache\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:44.873374 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.873347 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-home\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:44.875285 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.875244 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 14:19:44.875438 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.875323 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm" (OuterVolumeSpecName: "dshm") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:19:44.875438 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.875400 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp" (OuterVolumeSpecName: "kube-api-access-h2tnp") pod "2fb35bfd-1724-49ce-892a-51c18d6f7eec" (UID: "2fb35bfd-1724-49ce-892a-51c18d6f7eec"). InnerVolumeSpecName "kube-api-access-h2tnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:19:44.974038 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.973995 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2tnp\" (UniqueName: \"kubernetes.io/projected/2fb35bfd-1724-49ce-892a-51c18d6f7eec-kube-api-access-h2tnp\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:44.974038 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.974031 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tmp-dir\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:44.974038 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.974045 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb35bfd-1724-49ce-892a-51c18d6f7eec-tls-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:44.974280 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:44.974056 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb35bfd-1724-49ce-892a-51c18d6f7eec-dshm\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:19:45.214281 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.214186 2563 generic.go:356] "Generic (PLEG): container finished" podID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" containerID="3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528" exitCode=0 Jul 16 14:19:45.214281 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.214235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" Jul 16 14:19:45.214281 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.214243 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" event={"ID":"2fb35bfd-1724-49ce-892a-51c18d6f7eec","Type":"ContainerDied","Data":"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528"} Jul 16 14:19:45.214780 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.214286 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf" event={"ID":"2fb35bfd-1724-49ce-892a-51c18d6f7eec","Type":"ContainerDied","Data":"46817ba88a0e1a45fc95748341afa45228e45b6fcbed1b8b392856b66e84c9bd"} Jul 16 14:19:45.214780 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.214310 2563 scope.go:117] "RemoveContainer" containerID="3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528" Jul 16 14:19:45.224664 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.224583 2563 scope.go:117] "RemoveContainer" containerID="3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528" Jul 16 14:19:45.225062 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:19:45.225041 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528\": container with ID starting with 3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528 not found: ID does not exist" containerID="3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528" Jul 16 14:19:45.225124 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.225072 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528"} err="failed to get container status \"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528\": rpc error: code = NotFound desc = could not find container \"3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528\": container with ID starting with 3c75159b697dec7fac8120d0f3b7c41bc639886ff72e4cfd1ac0f1a6ec35a528 not found: ID does not exist" Jul 16 14:19:45.252893 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.252833 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:19:45.260850 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:45.260817 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/sub-test-model-1f53bd-kserve-5ff7895577-zmntf"] Jul 16 14:19:46.475235 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:19:46.475199 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" path="/var/lib/kubelet/pods/2fb35bfd-1724-49ce-892a-51c18d6f7eec/volumes" Jul 16 14:20:08.238619 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.238569 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:20:08.239358 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.239334 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" containerName="main" Jul 16 14:20:08.239431 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.239361 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" containerName="main" Jul 16 14:20:08.239489 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.239477 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb35bfd-1724-49ce-892a-51c18d6f7eec" containerName="main" Jul 16 14:20:08.243105 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.243085 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.246706 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.246677 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"default-dockercfg-7pkpp\"" Jul 16 14:20:08.248041 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.248022 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"rate-test-model-1f53bd-kserve-self-signed-certs\"" Jul 16 14:20:08.248625 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.248601 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"openshift-service-ca.crt\"" Jul 16 14:20:08.249048 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.249026 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f53bd\"/\"kube-root-ca.crt\"" Jul 16 14:20:08.258563 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.258539 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:20:08.284031 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.283993 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:08.399572 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399533 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.399572 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.399776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jsn\" (UniqueName: \"kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.399776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.399776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.399776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.399737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501276 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jsn\" (UniqueName: \"kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501695 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501695 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501632 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501820 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501690 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.501866 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.501848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.503571 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.503548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.503778 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.503760 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.511164 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.511145 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jsn\" (UniqueName: \"kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn\") pod \"rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.560395 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.560358 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:08.706404 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:08.706380 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:20:08.709077 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:20:08.709049 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf789ae71_2525_4c82_88a0_fdeed11e7ee3.slice/crio-e99f6127180474c23cccf6b7f5d8cb51888eff9ea880f9d390853013ac772109 WatchSource:0}: Error finding container e99f6127180474c23cccf6b7f5d8cb51888eff9ea880f9d390853013ac772109: Status 404 returned error can't find the container with id e99f6127180474c23cccf6b7f5d8cb51888eff9ea880f9d390853013ac772109 Jul 16 14:20:09.369867 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:09.369833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" event={"ID":"f789ae71-2525-4c82-88a0-fdeed11e7ee3","Type":"ContainerStarted","Data":"d2ad61fc1ee98419a9f62ec0680442ff177138b90ee45a717339a9185d749eb9"} Jul 16 14:20:09.370289 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:09.369891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" event={"ID":"f789ae71-2525-4c82-88a0-fdeed11e7ee3","Type":"ContainerStarted","Data":"e99f6127180474c23cccf6b7f5d8cb51888eff9ea880f9d390853013ac772109"} Jul 16 14:20:09.388633 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:09.388578 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" podStartSLOduration=1.388561301 podStartE2EDuration="1.388561301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:20:09.387374738 +0000 UTC m=+2389.335491445" watchObservedRunningTime="2026-07-16 14:20:09.388561301 +0000 UTC m=+2389.336678006" Jul 16 14:20:13.204325 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:13.204288 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:20:14.382745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:14.382705 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:14.483242 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:14.483207 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:18.560705 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:18.560668 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:18.561137 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:18.560724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:18.573422 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:18.573396 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:19.465928 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:19.465888 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:20:20.895793 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:20.895769 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:20:20.906538 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:20.906505 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ltrtt_9e789ac3-b4c4-42d6-85e4-c0dad9024b5e/ovn-acl-logging/0.log" Jul 16 14:20:22.558656 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:22.558622 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:22.647075 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:22.647042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:35.000843 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:35.000798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:20:37.305329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:37.305292 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:52.307965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:52.307870 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:52.404605 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:52.404565 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:20:54.322719 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:20:54.322682 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:21:00.834347 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:00.834306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:01.621847 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:01.621809 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:21:01.649668 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:01.649626 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:02.675503 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.675464 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567466003888"] Jul 16 14:21:02.679071 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.679050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:02.683417 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.683383 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gshn4\"" Jul 16 14:21:02.686897 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.686854 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567466003888"] Jul 16 14:21:02.731834 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.731799 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdcf\" (UniqueName: \"kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf\") pod \"test-curl-7789-140567466003888\" (UID: \"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d\") " pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:02.832927 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.832865 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdcf\" (UniqueName: \"kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf\") pod \"test-curl-7789-140567466003888\" (UID: \"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d\") " pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:02.846473 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.846438 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdcf\" (UniqueName: \"kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf\") pod \"test-curl-7789-140567466003888\" (UID: \"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d\") " pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:02.990236 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:02.990134 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:03.134056 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:03.134029 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567466003888"] Jul 16 14:21:03.136903 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:03.136858 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72e3b3b2_e131_4f0d_9d74_a7bea0b0ee7d.slice/crio-b19692f53b1a98a83e06b969511dd5a0a91fe451a1667ddb441afe1cf2d3e039 WatchSource:0}: Error finding container b19692f53b1a98a83e06b969511dd5a0a91fe451a1667ddb441afe1cf2d3e039: Status 404 returned error can't find the container with id b19692f53b1a98a83e06b969511dd5a0a91fe451a1667ddb441afe1cf2d3e039 Jul 16 14:21:03.138122 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:03.138097 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 14:21:03.752745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:03.752714 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567466003888" event={"ID":"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d","Type":"ContainerStarted","Data":"b19692f53b1a98a83e06b969511dd5a0a91fe451a1667ddb441afe1cf2d3e039"} Jul 16 14:21:04.760152 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:04.760114 2563 generic.go:356] "Generic (PLEG): container finished" podID="72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" containerID="bd6cc90ba08501eadf3a0f32a66660879873f0d76c2b67f37c62745b98dfe041" exitCode=0 Jul 16 14:21:04.760544 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:04.760158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567466003888" event={"ID":"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d","Type":"ContainerDied","Data":"bd6cc90ba08501eadf3a0f32a66660879873f0d76c2b67f37c62745b98dfe041"} Jul 16 14:21:05.905649 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.905614 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:05.935125 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.935096 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567466003888_72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d/test-curl-7789-140567466003888/0.log" Jul 16 14:21:05.965561 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.965529 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vdcf\" (UniqueName: \"kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf\") pod \"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d\" (UID: \"72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d\") " Jul 16 14:21:05.967792 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.967754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf" (OuterVolumeSpecName: "kube-api-access-2vdcf") pod "72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" (UID: "72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d"). InnerVolumeSpecName "kube-api-access-2vdcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:05.978068 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.978042 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567466003888"] Jul 16 14:21:05.984103 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:05.984081 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567466003888"] Jul 16 14:21:06.066554 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.066446 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdcf\" (UniqueName: \"kubernetes.io/projected/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d-kube-api-access-2vdcf\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:06.122128 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.122089 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567466000432"] Jul 16 14:21:06.122534 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.122521 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" containerName="test-curl-7789-140567466003888" Jul 16 14:21:06.122578 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.122535 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" containerName="test-curl-7789-140567466003888" Jul 16 14:21:06.122654 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.122644 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" containerName="test-curl-7789-140567466003888" Jul 16 14:21:06.126013 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.125995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:06.131970 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.131943 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567466000432"] Jul 16 14:21:06.167418 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.167384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnwt\" (UniqueName: \"kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt\") pod \"test-curl-7789-140567466000432\" (UID: \"948c22ef-4bf7-4e69-8dac-825145f3e532\") " pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:06.268363 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.268321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnwt\" (UniqueName: \"kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt\") pod \"test-curl-7789-140567466000432\" (UID: \"948c22ef-4bf7-4e69-8dac-825145f3e532\") " pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:06.277159 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.277120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnwt\" (UniqueName: \"kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt\") pod \"test-curl-7789-140567466000432\" (UID: \"948c22ef-4bf7-4e69-8dac-825145f3e532\") " pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:06.436924 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.436812 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:06.475737 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.475708 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d" path="/var/lib/kubelet/pods/72e3b3b2-e131-4f0d-9d74-a7bea0b0ee7d/volumes" Jul 16 14:21:06.591603 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.591575 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567466000432"] Jul 16 14:21:06.593076 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:06.593038 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod948c22ef_4bf7_4e69_8dac_825145f3e532.slice/crio-113417ccb88d010680f5a2f9697174bd6914014e13e8334289c1e06fa14db6e9 WatchSource:0}: Error finding container 113417ccb88d010680f5a2f9697174bd6914014e13e8334289c1e06fa14db6e9: Status 404 returned error can't find the container with id 113417ccb88d010680f5a2f9697174bd6914014e13e8334289c1e06fa14db6e9 Jul 16 14:21:06.774965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.774926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567466000432" event={"ID":"948c22ef-4bf7-4e69-8dac-825145f3e532","Type":"ContainerStarted","Data":"113417ccb88d010680f5a2f9697174bd6914014e13e8334289c1e06fa14db6e9"} Jul 16 14:21:06.776133 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.776117 2563 scope.go:117] "RemoveContainer" containerID="bd6cc90ba08501eadf3a0f32a66660879873f0d76c2b67f37c62745b98dfe041" Jul 16 14:21:06.776212 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:06.776142 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466003888" Jul 16 14:21:07.785065 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:07.785030 2563 generic.go:356] "Generic (PLEG): container finished" podID="948c22ef-4bf7-4e69-8dac-825145f3e532" containerID="fd970ed550920ee3c36bdd3f026bd0764e14fc6d3035020082adbfac1fe13eac" exitCode=0 Jul 16 14:21:07.785524 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:07.785116 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567466000432" event={"ID":"948c22ef-4bf7-4e69-8dac-825145f3e532","Type":"ContainerDied","Data":"fd970ed550920ee3c36bdd3f026bd0764e14fc6d3035020082adbfac1fe13eac"} Jul 16 14:21:08.944585 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:08.944555 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:08.967746 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:08.967721 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567466000432_948c22ef-4bf7-4e69-8dac-825145f3e532/test-curl-7789-140567466000432/0.log" Jul 16 14:21:08.993924 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:08.993886 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wnwt\" (UniqueName: \"kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt\") pod \"948c22ef-4bf7-4e69-8dac-825145f3e532\" (UID: \"948c22ef-4bf7-4e69-8dac-825145f3e532\") " Jul 16 14:21:09.004972 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.004931 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567466000432"] Jul 16 14:21:09.005225 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.005124 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt" (OuterVolumeSpecName: "kube-api-access-2wnwt") pod "948c22ef-4bf7-4e69-8dac-825145f3e532" (UID: "948c22ef-4bf7-4e69-8dac-825145f3e532"). InnerVolumeSpecName "kube-api-access-2wnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:09.006683 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.006660 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567466000432"] Jul 16 14:21:09.095125 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.095033 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wnwt\" (UniqueName: \"kubernetes.io/projected/948c22ef-4bf7-4e69-8dac-825145f3e532-kube-api-access-2wnwt\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:09.178222 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.178185 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:09.178666 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.178652 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" containerName="test-curl-7789-140567466000432" Jul 16 14:21:09.178709 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.178668 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" containerName="test-curl-7789-140567466000432" Jul 16 14:21:09.178749 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.178736 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" containerName="test-curl-7789-140567466000432" Jul 16 14:21:09.181948 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.181932 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:09.190332 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.190300 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:09.296710 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.296646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gprl7\" (UniqueName: \"kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7\") pod \"test-curl-7789-140567462944320\" (UID: \"e78042df-6564-4d57-95f8-075ebcf59e7b\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:09.398030 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.397930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprl7\" (UniqueName: \"kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7\") pod \"test-curl-7789-140567462944320\" (UID: \"e78042df-6564-4d57-95f8-075ebcf59e7b\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:09.408250 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.408223 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprl7\" (UniqueName: \"kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7\") pod \"test-curl-7789-140567462944320\" (UID: \"e78042df-6564-4d57-95f8-075ebcf59e7b\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:09.493005 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.492973 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:09.644261 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.644230 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:09.645396 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:09.645365 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode78042df_6564_4d57_95f8_075ebcf59e7b.slice/crio-cd7037678770162ce512b5d57d3fd46e87760d2de67cbd3e15092926669f1923 WatchSource:0}: Error finding container cd7037678770162ce512b5d57d3fd46e87760d2de67cbd3e15092926669f1923: Status 404 returned error can't find the container with id cd7037678770162ce512b5d57d3fd46e87760d2de67cbd3e15092926669f1923 Jul 16 14:21:09.799849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.799819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567462944320" event={"ID":"e78042df-6564-4d57-95f8-075ebcf59e7b","Type":"ContainerStarted","Data":"cd7037678770162ce512b5d57d3fd46e87760d2de67cbd3e15092926669f1923"} Jul 16 14:21:09.801181 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.801161 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="113417ccb88d010680f5a2f9697174bd6914014e13e8334289c1e06fa14db6e9" Jul 16 14:21:09.801329 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:09.801204 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:09.812746 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:21:09.812713 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-140567466000432\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:10.102673 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:10.102630 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:21:10.474054 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:21:10.474015 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-140567466000432\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" pod="opendatahub/test-curl-7789-140567466000432" Jul 16 14:21:10.475545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:10.475520 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948c22ef-4bf7-4e69-8dac-825145f3e532" path="/var/lib/kubelet/pods/948c22ef-4bf7-4e69-8dac-825145f3e532/volumes" Jul 16 14:21:10.809438 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:10.809403 2563 generic.go:356] "Generic (PLEG): container finished" podID="e78042df-6564-4d57-95f8-075ebcf59e7b" containerID="7a9dcc24d0bdf7a33e3591c480c93dd8186b4959f3f84544717cd84312e36f79" exitCode=0 Jul 16 14:21:10.809626 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:10.809487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567462944320" event={"ID":"e78042df-6564-4d57-95f8-075ebcf59e7b","Type":"ContainerDied","Data":"7a9dcc24d0bdf7a33e3591c480c93dd8186b4959f3f84544717cd84312e36f79"} Jul 16 14:21:11.960829 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:11.960799 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:11.980989 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:11.980961 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567462944320_e78042df-6564-4d57-95f8-075ebcf59e7b/test-curl-7789-140567462944320/0.log" Jul 16 14:21:12.006260 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.006216 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:12.009887 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.009855 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:12.021849 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.021821 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gprl7\" (UniqueName: \"kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7\") pod \"e78042df-6564-4d57-95f8-075ebcf59e7b\" (UID: \"e78042df-6564-4d57-95f8-075ebcf59e7b\") " Jul 16 14:21:12.023944 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.023912 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7" (OuterVolumeSpecName: "kube-api-access-gprl7") pod "e78042df-6564-4d57-95f8-075ebcf59e7b" (UID: "e78042df-6564-4d57-95f8-075ebcf59e7b"). InnerVolumeSpecName "kube-api-access-gprl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:12.122852 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.122756 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gprl7\" (UniqueName: \"kubernetes.io/projected/e78042df-6564-4d57-95f8-075ebcf59e7b-kube-api-access-gprl7\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:12.135711 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.135677 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:12.136134 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.136120 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78042df-6564-4d57-95f8-075ebcf59e7b" containerName="test-curl-7789-140567462944320" Jul 16 14:21:12.136193 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.136136 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78042df-6564-4d57-95f8-075ebcf59e7b" containerName="test-curl-7789-140567462944320" Jul 16 14:21:12.136236 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.136227 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78042df-6564-4d57-95f8-075ebcf59e7b" containerName="test-curl-7789-140567462944320" Jul 16 14:21:12.139339 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.139321 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:12.147945 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.147922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:12.223666 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.223627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7w8\" (UniqueName: \"kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8\") pod \"test-curl-7789-140567462944320\" (UID: \"7c645389-7dc8-4cea-83b8-db17038b06a4\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:12.324244 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.324202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7w8\" (UniqueName: \"kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8\") pod \"test-curl-7789-140567462944320\" (UID: \"7c645389-7dc8-4cea-83b8-db17038b06a4\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:12.336615 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.336581 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7w8\" (UniqueName: \"kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8\") pod \"test-curl-7789-140567462944320\" (UID: \"7c645389-7dc8-4cea-83b8-db17038b06a4\") " pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:12.451173 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.451068 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:12.482869 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.482637 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78042df-6564-4d57-95f8-075ebcf59e7b" path="/var/lib/kubelet/pods/e78042df-6564-4d57-95f8-075ebcf59e7b/volumes" Jul 16 14:21:12.604247 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.604217 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:12.605292 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:12.605264 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c645389_7dc8_4cea_83b8_db17038b06a4.slice/crio-ea14af8f0430edccbf389bbfeb3968de56324a0603f1e792d3dd2a608fa3171e WatchSource:0}: Error finding container ea14af8f0430edccbf389bbfeb3968de56324a0603f1e792d3dd2a608fa3171e: Status 404 returned error can't find the container with id ea14af8f0430edccbf389bbfeb3968de56324a0603f1e792d3dd2a608fa3171e Jul 16 14:21:12.823346 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.823307 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567462944320" event={"ID":"7c645389-7dc8-4cea-83b8-db17038b06a4","Type":"ContainerStarted","Data":"ea14af8f0430edccbf389bbfeb3968de56324a0603f1e792d3dd2a608fa3171e"} Jul 16 14:21:12.824698 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.824679 2563 scope.go:117] "RemoveContainer" containerID="7a9dcc24d0bdf7a33e3591c480c93dd8186b4959f3f84544717cd84312e36f79" Jul 16 14:21:12.824834 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:12.824710 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:13.834446 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:13.834413 2563 generic.go:356] "Generic (PLEG): container finished" podID="7c645389-7dc8-4cea-83b8-db17038b06a4" containerID="4508ae2deff3a9375318bf5a9e14a7633ba03b717b97bfd7be46718dd37ff5a6" exitCode=0 Jul 16 14:21:13.834856 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:13.834479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567462944320" event={"ID":"7c645389-7dc8-4cea-83b8-db17038b06a4","Type":"ContainerDied","Data":"4508ae2deff3a9375318bf5a9e14a7633ba03b717b97bfd7be46718dd37ff5a6"} Jul 16 14:21:14.988200 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:14.988177 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:15.009950 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.009920 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567462944320_7c645389-7dc8-4cea-83b8-db17038b06a4/test-curl-7789-140567462944320/0.log" Jul 16 14:21:15.038781 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.038749 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:15.043211 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.043180 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567462944320"] Jul 16 14:21:15.051814 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.051787 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s7w8\" (UniqueName: \"kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8\") pod \"7c645389-7dc8-4cea-83b8-db17038b06a4\" (UID: \"7c645389-7dc8-4cea-83b8-db17038b06a4\") " Jul 16 14:21:15.053919 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.053893 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8" (OuterVolumeSpecName: "kube-api-access-9s7w8") pod "7c645389-7dc8-4cea-83b8-db17038b06a4" (UID: "7c645389-7dc8-4cea-83b8-db17038b06a4"). InnerVolumeSpecName "kube-api-access-9s7w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:15.153145 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.153056 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7w8\" (UniqueName: \"kubernetes.io/projected/7c645389-7dc8-4cea-83b8-db17038b06a4-kube-api-access-9s7w8\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:15.184803 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.184767 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:15.185254 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.185240 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c645389-7dc8-4cea-83b8-db17038b06a4" containerName="test-curl-7789-140567462944320" Jul 16 14:21:15.185315 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.185257 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c645389-7dc8-4cea-83b8-db17038b06a4" containerName="test-curl-7789-140567462944320" Jul 16 14:21:15.185357 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.185318 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c645389-7dc8-4cea-83b8-db17038b06a4" containerName="test-curl-7789-140567462944320" Jul 16 14:21:15.188604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.188587 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:15.195507 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.195482 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:15.254173 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.254133 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fmk\" (UniqueName: \"kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk\") pod \"test-curl-7789-140567468966512\" (UID: \"da9ed360-682f-4886-a1da-a9acb1aac0a3\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:15.354889 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.354848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fmk\" (UniqueName: \"kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk\") pod \"test-curl-7789-140567468966512\" (UID: \"da9ed360-682f-4886-a1da-a9acb1aac0a3\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:15.363452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.363413 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fmk\" (UniqueName: \"kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk\") pod \"test-curl-7789-140567468966512\" (UID: \"da9ed360-682f-4886-a1da-a9acb1aac0a3\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:15.498989 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.498866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:15.639288 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.639256 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:15.640347 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:15.640317 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9ed360_682f_4886_a1da_a9acb1aac0a3.slice/crio-106f9a7f11028a66c4fa9de45d0654b6223f7bc2a2e5b7dc029586d91190f448 WatchSource:0}: Error finding container 106f9a7f11028a66c4fa9de45d0654b6223f7bc2a2e5b7dc029586d91190f448: Status 404 returned error can't find the container with id 106f9a7f11028a66c4fa9de45d0654b6223f7bc2a2e5b7dc029586d91190f448 Jul 16 14:21:15.847580 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.847537 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567468966512" event={"ID":"da9ed360-682f-4886-a1da-a9acb1aac0a3","Type":"ContainerStarted","Data":"106f9a7f11028a66c4fa9de45d0654b6223f7bc2a2e5b7dc029586d91190f448"} Jul 16 14:21:15.848907 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.848887 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea14af8f0430edccbf389bbfeb3968de56324a0603f1e792d3dd2a608fa3171e" Jul 16 14:21:15.849037 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:15.848931 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:15.860348 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:21:15.860320 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-140567462944320\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="7c645389-7dc8-4cea-83b8-db17038b06a4" pod="opendatahub/test-curl-7789-140567462944320" Jul 16 14:21:16.475315 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.475280 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c645389-7dc8-4cea-83b8-db17038b06a4" path="/var/lib/kubelet/pods/7c645389-7dc8-4cea-83b8-db17038b06a4/volumes" Jul 16 14:21:16.643803 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.643745 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" podUID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" containerName="main" containerID="cri-o://d2ad61fc1ee98419a9f62ec0680442ff177138b90ee45a717339a9185d749eb9" gracePeriod=45 Jul 16 14:21:16.857247 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.857213 2563 generic.go:356] "Generic (PLEG): container finished" podID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" containerID="d2ad61fc1ee98419a9f62ec0680442ff177138b90ee45a717339a9185d749eb9" exitCode=0 Jul 16 14:21:16.857403 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.857285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" event={"ID":"f789ae71-2525-4c82-88a0-fdeed11e7ee3","Type":"ContainerDied","Data":"d2ad61fc1ee98419a9f62ec0680442ff177138b90ee45a717339a9185d749eb9"} Jul 16 14:21:16.859069 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.859048 2563 generic.go:356] "Generic (PLEG): container finished" podID="da9ed360-682f-4886-a1da-a9acb1aac0a3" containerID="9f9c03049ff5ad48ccd12a0072a3e7eb56f3169a5f3b8c404aa46b5a18c3fb71" exitCode=0 Jul 16 14:21:16.859164 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.859132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567468966512" event={"ID":"da9ed360-682f-4886-a1da-a9acb1aac0a3","Type":"ContainerDied","Data":"9f9c03049ff5ad48ccd12a0072a3e7eb56f3169a5f3b8c404aa46b5a18c3fb71"} Jul 16 14:21:16.914940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.914912 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:21:16.970310 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970272 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970310 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970313 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970549 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55jsn\" (UniqueName: \"kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970549 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970419 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970549 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970446 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970549 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970466 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home\") pod \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\" (UID: \"f789ae71-2525-4c82-88a0-fdeed11e7ee3\") " Jul 16 14:21:16.970773 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970701 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:16.970773 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970716 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:16.970931 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.970904 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home" (OuterVolumeSpecName: "home") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:16.972585 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.972550 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 14:21:16.972702 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.972631 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm" (OuterVolumeSpecName: "dshm") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:16.972702 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:16.972635 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn" (OuterVolumeSpecName: "kube-api-access-55jsn") pod "f789ae71-2525-4c82-88a0-fdeed11e7ee3" (UID: "f789ae71-2525-4c82-88a0-fdeed11e7ee3"). InnerVolumeSpecName "kube-api-access-55jsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:17.071960 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.071917 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-home\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.071960 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.071952 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tls-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.071960 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.071966 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-tmp-dir\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.072224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.071979 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jsn\" (UniqueName: \"kubernetes.io/projected/f789ae71-2525-4c82-88a0-fdeed11e7ee3-kube-api-access-55jsn\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.072224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.071992 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-model-cache\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.072224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.072002 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f789ae71-2525-4c82-88a0-fdeed11e7ee3-dshm\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:17.867776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.867748 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" Jul 16 14:21:17.867776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.867759 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc" event={"ID":"f789ae71-2525-4c82-88a0-fdeed11e7ee3","Type":"ContainerDied","Data":"e99f6127180474c23cccf6b7f5d8cb51888eff9ea880f9d390853013ac772109"} Jul 16 14:21:17.868335 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.867809 2563 scope.go:117] "RemoveContainer" containerID="d2ad61fc1ee98419a9f62ec0680442ff177138b90ee45a717339a9185d749eb9" Jul 16 14:21:17.892069 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.892036 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:21:17.896359 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:17.896330 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1f53bd/rate-test-model-1f53bd-kserve-b9cc7d98c-r7lsc"] Jul 16 14:21:18.017100 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.017069 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.040608 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.040573 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567468966512_da9ed360-682f-4886-a1da-a9acb1aac0a3/test-curl-7789-140567468966512/0.log" Jul 16 14:21:18.069810 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.069777 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:18.072526 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.072504 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:18.082224 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.082200 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6fmk\" (UniqueName: \"kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk\") pod \"da9ed360-682f-4886-a1da-a9acb1aac0a3\" (UID: \"da9ed360-682f-4886-a1da-a9acb1aac0a3\") " Jul 16 14:21:18.084244 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.084213 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk" (OuterVolumeSpecName: "kube-api-access-z6fmk") pod "da9ed360-682f-4886-a1da-a9acb1aac0a3" (UID: "da9ed360-682f-4886-a1da-a9acb1aac0a3"). InnerVolumeSpecName "kube-api-access-z6fmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:18.182945 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.182847 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6fmk\" (UniqueName: \"kubernetes.io/projected/da9ed360-682f-4886-a1da-a9acb1aac0a3-kube-api-access-z6fmk\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:18.210729 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.210691 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:18.211445 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211423 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" containerName="main" Jul 16 14:21:18.211489 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211450 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" containerName="main" Jul 16 14:21:18.211521 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211492 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9ed360-682f-4886-a1da-a9acb1aac0a3" containerName="test-curl-7789-140567468966512" Jul 16 14:21:18.211521 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211502 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9ed360-682f-4886-a1da-a9acb1aac0a3" containerName="test-curl-7789-140567468966512" Jul 16 14:21:18.211619 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211607 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" containerName="main" Jul 16 14:21:18.211652 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.211628 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9ed360-682f-4886-a1da-a9acb1aac0a3" containerName="test-curl-7789-140567468966512" Jul 16 14:21:18.215414 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.215398 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.222765 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.222740 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:18.284241 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.284199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5rj\" (UniqueName: \"kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj\") pod \"test-curl-7789-140567468966512\" (UID: \"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.385680 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.385641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5rj\" (UniqueName: \"kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj\") pod \"test-curl-7789-140567468966512\" (UID: \"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.395046 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.395015 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5rj\" (UniqueName: \"kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj\") pod \"test-curl-7789-140567468966512\" (UID: \"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4\") " pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.475151 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.475065 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9ed360-682f-4886-a1da-a9acb1aac0a3" path="/var/lib/kubelet/pods/da9ed360-682f-4886-a1da-a9acb1aac0a3/volumes" Jul 16 14:21:18.475492 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.475478 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f789ae71-2525-4c82-88a0-fdeed11e7ee3" path="/var/lib/kubelet/pods/f789ae71-2525-4c82-88a0-fdeed11e7ee3/volumes" Jul 16 14:21:18.525410 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.525361 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.675428 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:18.675399 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode855942f_16fe_4b77_a9d2_2d4fb8ed71b4.slice/crio-43bc3ebf0c437370cb676347df4283c114f9c66c1e7385d391b058ccd2601c53 WatchSource:0}: Error finding container 43bc3ebf0c437370cb676347df4283c114f9c66c1e7385d391b058ccd2601c53: Status 404 returned error can't find the container with id 43bc3ebf0c437370cb676347df4283c114f9c66c1e7385d391b058ccd2601c53 Jul 16 14:21:18.676001 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.675976 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:18.875261 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.875222 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567468966512" event={"ID":"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4","Type":"ContainerStarted","Data":"43bc3ebf0c437370cb676347df4283c114f9c66c1e7385d391b058ccd2601c53"} Jul 16 14:21:18.877348 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.877327 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:18.877464 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:18.877382 2563 scope.go:117] "RemoveContainer" containerID="9f9c03049ff5ad48ccd12a0072a3e7eb56f3169a5f3b8c404aa46b5a18c3fb71" Jul 16 14:21:19.885392 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:19.885360 2563 generic.go:356] "Generic (PLEG): container finished" podID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" containerID="abb543a7bc01f09f20ea39c2707a3cdad99a0d7c24a2ce329836ee2a2665c12f" exitCode=0 Jul 16 14:21:19.885828 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:19.885446 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-140567468966512" event={"ID":"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4","Type":"ContainerDied","Data":"abb543a7bc01f09f20ea39c2707a3cdad99a0d7c24a2ce329836ee2a2665c12f"} Jul 16 14:21:21.036614 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.036590 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:21.068865 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.068831 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7789-140567468966512_e855942f-16fe-4b77-a9d2-2d4fb8ed71b4/test-curl-7789-140567468966512/0.log" Jul 16 14:21:21.094787 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.094754 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:21.098742 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.098713 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-140567468966512"] Jul 16 14:21:21.112278 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.112249 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r5rj\" (UniqueName: \"kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj\") pod \"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4\" (UID: \"e855942f-16fe-4b77-a9d2-2d4fb8ed71b4\") " Jul 16 14:21:21.114380 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.114354 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj" (OuterVolumeSpecName: "kube-api-access-6r5rj") pod "e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" (UID: "e855942f-16fe-4b77-a9d2-2d4fb8ed71b4"). InnerVolumeSpecName "kube-api-access-6r5rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:21.213426 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.213335 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5rj\" (UniqueName: \"kubernetes.io/projected/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4-kube-api-access-6r5rj\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:21.689750 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.689708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:21.900328 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.900300 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:21.900499 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:21.900306 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bc3ebf0c437370cb676347df4283c114f9c66c1e7385d391b058ccd2601c53" Jul 16 14:21:21.910318 ip-10-0-130-14 kubenswrapper[2563]: E0716 14:21:21.910279 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-140567468966512\" is forbidden: User \"system:node:ip-10-0-130-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-14.ec2.internal' and this object" podUID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" pod="opendatahub/test-curl-7789-140567468966512" Jul 16 14:21:22.475155 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:22.475123 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" path="/var/lib/kubelet/pods/e855942f-16fe-4b77-a9d2-2d4fb8ed71b4/volumes" Jul 16 14:21:26.768105 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:26.768065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:28.519205 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:28.519174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:28.623495 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:28.623461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:28.828782 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:28.828685 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:21:28.829002 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:28.828976 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" containerID="cri-o://0744715aae1b5458883afc598b71ee031a8314bc3b91efffbf10786e854831f9" gracePeriod=30 Jul 16 14:21:28.852197 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:28.852159 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:29.087295 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:29.087191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:29.612287 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:29.612249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:32.089723 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:32.089683 2563 patch_prober.go:28] interesting pod/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 14:21:32.090121 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:32.089757 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 14:21:33.219156 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:33.219115 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:21:33.987457 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:33.987426 2563 generic.go:356] "Generic (PLEG): container finished" podID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerID="0744715aae1b5458883afc598b71ee031a8314bc3b91efffbf10786e854831f9" exitCode=0 Jul 16 14:21:33.987626 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:33.987503 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerDied","Data":"0744715aae1b5458883afc598b71ee031a8314bc3b91efffbf10786e854831f9"} Jul 16 14:21:33.987626 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:33.987570 2563 scope.go:117] "RemoveContainer" containerID="ba60f21399135ed39b18ccf2fe88a2080a150c28d7e274d53474bb554b2e1e28" Jul 16 14:21:34.098965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.098891 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:21:34.238536 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238504 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.238536 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238542 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238562 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238647 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238698 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238776 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4ljh\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238801 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238850 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo\") pod \"b339ab84-e9ae-4c14-a8ac-9c850383b388\" (UID: \"b339ab84-e9ae-4c14-a8ac-9c850383b388\") " Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.238933 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:34.239082 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239064 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239140 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239273 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239291 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-workload-certs\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239308 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-credential-socket\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239384 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 14:21:34.239562 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.239432 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data" (OuterVolumeSpecName: "istio-data") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:34.241220 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.241189 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh" (OuterVolumeSpecName: "kube-api-access-m4ljh") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "kube-api-access-m4ljh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:34.241363 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.241340 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token" (OuterVolumeSpecName: "istio-token") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 14:21:34.241488 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.241458 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 14:21:34.241629 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.241608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b339ab84-e9ae-4c14-a8ac-9c850383b388" (UID: "b339ab84-e9ae-4c14-a8ac-9c850383b388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 14:21:34.340538 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340504 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-envoy\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.340538 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340532 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b339ab84-e9ae-4c14-a8ac-9c850383b388-istiod-ca-cert\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.340538 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340542 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-token\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.340790 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340550 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ljh\" (UniqueName: \"kubernetes.io/projected/b339ab84-e9ae-4c14-a8ac-9c850383b388-kube-api-access-m4ljh\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.340790 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340560 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-podinfo\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.340790 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.340567 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b339ab84-e9ae-4c14-a8ac-9c850383b388-istio-data\") on node \"ip-10-0-130-14.ec2.internal\" DevicePath \"\"" Jul 16 14:21:34.995339 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.995295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" event={"ID":"b339ab84-e9ae-4c14-a8ac-9c850383b388","Type":"ContainerDied","Data":"24bda3c53047f2c93286a01c628797d17132c4cfecd9a6ff2c17c4071117ba1f"} Jul 16 14:21:34.995664 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.995646 2563 scope.go:117] "RemoveContainer" containerID="0744715aae1b5458883afc598b71ee031a8314bc3b91efffbf10786e854831f9" Jul 16 14:21:34.995913 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:34.995403 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh" Jul 16 14:21:35.018868 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:35.018835 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:21:35.026503 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:35.026473 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c8253c-openshift-default-7f5b56f759-wlqgh"] Jul 16 14:21:35.671983 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:35.671954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:35.772569 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:35.772530 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:35.961823 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:35.961735 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:36.198176 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:36.198133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:36.474558 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:36.474523 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" path="/var/lib/kubelet/pods/b339ab84-e9ae-4c14-a8ac-9c850383b388/volumes" Jul 16 14:21:36.717798 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:36.717765 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564fb575c6-jxlnd_4578272c-787f-420f-8a70-3d5e08fa9bea/authorino/0.log" Jul 16 14:21:43.046272 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:43.046243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxkwr_7e77e136-2d2c-40a8-b152-7485f4a6339b/server/0.log" Jul 16 14:21:43.182423 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:43.182389 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fctwd_bb228b98-3ec5-4802-a27e-8993fa0c68e9/manager/0.log" Jul 16 14:21:43.458376 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:43.458297 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-558d6dc665-7fzrq_c06c9634-bc6c-4670-8311-47245d2731ba/copy-manifests/0.log" Jul 16 14:21:43.466934 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:43.466904 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-558d6dc665-7fzrq_c06c9634-bc6c-4670-8311-47245d2731ba/manager/0.log" Jul 16 14:21:45.525783 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:45.525745 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7w8bs"] Jul 16 14:21:45.589317 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:45.589291 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736855-w4fd8_937765e1-5368-4c8c-898f-1335c79c192b/cleanup/0.log" Jul 16 14:21:45.720768 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:45.720736 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-66781e21-29736855-whm7j_5333d179-87cd-43cd-853a-8b0f8af23967/cleanup/0.log" Jul 16 14:21:45.848051 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:45.847963 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-a-1305fb-29736855-h77gg_743a2269-8aff-4cee-b363-9ff832c906af/cleanup/0.log" Jul 16 14:21:45.969685 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:45.969653 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-b-bb5499-29736855-lp7wd_0ee14a4f-87cb-4dd9-811e-acc612c05579/cleanup/0.log" Jul 16 14:21:46.095148 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.095115 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-derive-a7066c88-29736855-rtj22_ab5ad6b7-96f7-4d73-ac34-dd574c69fd8d/cleanup/0.log" Jul 16 14:21:46.219463 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.219373 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-6cdc6b-29736855-8khvn_28c434d0-a416-4e1a-a8c1-7a41d2db086f/cleanup/0.log" Jul 16 14:21:46.345901 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.345863 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-84f3c8-29736855-mvhcj_4fcfd902-58b5-45a0-9269-a2f8c5cad5e5/cleanup/0.log" Jul 16 14:21:46.470239 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.470148 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-migrate-8fabccd0-29736855-bxrb2_c929ea50-377d-439b-9d05-13da74665e2f/cleanup/0.log" Jul 16 14:21:46.597720 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.597692 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-mt-49f7930e-29736855-l2psm_a0114c98-c8b5-4011-a57d-471ae4c6af1f/cleanup/0.log" Jul 16 14:21:46.723012 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.722918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-c8253c-29736855-w424t_9e753e93-c9f5-4a41-a4e8-fbe79c1f3206/cleanup/0.log" Jul 16 14:21:46.862181 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.862149 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-1f53bd-29736855-svlxv_4db68e6e-cab9-4fd7-ac1f-e1b949dda4cd/cleanup/1.log" Jul 16 14:21:46.986920 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:46.986817 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d48v7_c0cfeb2d-2092-451e-a468-c68dd62f3d18/postgres/0.log" Jul 16 14:21:47.853317 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:47.853285 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/util/0.log" Jul 16 14:21:47.862412 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:47.862379 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/pull/0.log" Jul 16 14:21:47.869203 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:47.869175 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/extract/0.log" Jul 16 14:21:47.992775 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:47.992743 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/extract/0.log" Jul 16 14:21:47.998864 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:47.998834 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/util/0.log" Jul 16 14:21:48.005240 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.005219 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/pull/0.log" Jul 16 14:21:48.134019 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.133935 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/util/0.log" Jul 16 14:21:48.141375 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.141340 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/pull/0.log" Jul 16 14:21:48.148586 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.148556 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/extract/0.log" Jul 16 14:21:48.273002 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.272968 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/util/0.log" Jul 16 14:21:48.280326 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.280305 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/pull/0.log" Jul 16 14:21:48.287060 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.287042 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/extract/0.log" Jul 16 14:21:48.422796 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.422713 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564fb575c6-jxlnd_4578272c-787f-420f-8a70-3d5e08fa9bea/authorino/0.log" Jul 16 14:21:48.552345 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.552304 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dljr_4ebf3195-b824-47e9-8b24-719b02824133/manager/0.log" Jul 16 14:21:48.679130 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.679043 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gzp4h_adc7174e-48ed-485e-b686-58461c6f7c9c/manager/0.log" Jul 16 14:21:48.803126 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.803099 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fs8np_70b52ffa-2f77-4ac6-bfab-9fcb27b3d425/kuadrant-console-plugin/0.log" Jul 16 14:21:48.938210 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:48.938121 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gkdd4_e8ba6a14-303c-4ef2-8125-e0b1872b3a62/registry-server/0.log" Jul 16 14:21:49.076676 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:49.076644 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q67hs_3d6c259d-1f12-4506-a925-0d1e77039971/manager/0.log" Jul 16 14:21:49.208336 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:49.208247 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7w8bs_9511c1dd-cfb2-494d-859b-908fa156e867/limitador/0.log" Jul 16 14:21:50.134263 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:50.134235 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b75db759-xwgcj_b7204850-f591-4438-8c51-ace01411191a/kube-auth-proxy/0.log" Jul 16 14:21:50.273249 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:50.273212 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/1.log" Jul 16 14:21:50.534367 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:50.534336 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-wswqx_b6224b78-f7ed-435e-9283-5fcf886e4e5d/payload-processing/0.log" Jul 16 14:21:50.665348 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:50.665323 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5845f855c8-qr8xb_a3383b24-2dfc-45d3-9480-855b6e22b92e/router/0.log" Jul 16 14:21:51.177523 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:51.177493 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bddjz_e21d7c36-a0fd-494a-a492-62e3d7bde22e/main/0.log" Jul 16 14:21:51.427685 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:51.427608 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm6k2r_ebdec5b6-6bdd-4b04-9aa0-7d91142605de/main/0.log" Jul 16 14:21:51.680655 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:51.680557 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-sw4h4_a3dff74b-97dc-4b26-b59b-78737cca3392/main/0.log" Jul 16 14:21:57.794764 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.794723 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mhhqm/must-gather-rxs4b"] Jul 16 14:21:57.795437 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795382 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" containerName="test-curl-7789-140567468966512" Jul 16 14:21:57.795437 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795404 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" containerName="test-curl-7789-140567468966512" Jul 16 14:21:57.795604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795439 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.795604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795448 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.795604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795569 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e855942f-16fe-4b77-a9d2-2d4fb8ed71b4" containerName="test-curl-7789-140567468966512" Jul 16 14:21:57.795604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795588 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.795604 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795599 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.795973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795723 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.795973 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.795735 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b339ab84-e9ae-4c14-a8ac-9c850383b388" containerName="istio-proxy" Jul 16 14:21:57.798831 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.798810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.801545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.801519 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mhhqm\"/\"kube-root-ca.crt\"" Jul 16 14:21:57.801695 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.801519 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mhhqm\"/\"openshift-service-ca.crt\"" Jul 16 14:21:57.802406 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.802380 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mhhqm\"/\"default-dockercfg-6lmwq\"" Jul 16 14:21:57.810745 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.810720 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mhhqm/must-gather-rxs4b"] Jul 16 14:21:57.824505 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.824459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:57.857055 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.857007 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed378841-7aa5-4141-852f-b0cfbf784049-must-gather-output\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.857248 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.857066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprtq\" (UniqueName: \"kubernetes.io/projected/ed378841-7aa5-4141-852f-b0cfbf784049-kube-api-access-rprtq\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.904526 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.904487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" limit=1024 Jul 16 14:21:57.958452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.958414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed378841-7aa5-4141-852f-b0cfbf784049-must-gather-output\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.958452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.958458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprtq\" (UniqueName: \"kubernetes.io/projected/ed378841-7aa5-4141-852f-b0cfbf784049-kube-api-access-rprtq\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.958839 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.958815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed378841-7aa5-4141-852f-b0cfbf784049-must-gather-output\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:57.975420 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:57.975383 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprtq\" (UniqueName: \"kubernetes.io/projected/ed378841-7aa5-4141-852f-b0cfbf784049-kube-api-access-rprtq\") pod \"must-gather-rxs4b\" (UID: \"ed378841-7aa5-4141-852f-b0cfbf784049\") " pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:58.110940 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:58.110811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" Jul 16 14:21:58.264967 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:58.264940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mhhqm/must-gather-rxs4b"] Jul 16 14:21:58.266667 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:21:58.266627 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded378841_7aa5_4141_852f_b0cfbf784049.slice/crio-27238231dd67269f40b15e556f61fa9c6956fcda78b6a739abb12b124fa50e3c WatchSource:0}: Error finding container 27238231dd67269f40b15e556f61fa9c6956fcda78b6a739abb12b124fa50e3c: Status 404 returned error can't find the container with id 27238231dd67269f40b15e556f61fa9c6956fcda78b6a739abb12b124fa50e3c Jul 16 14:21:59.155176 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:21:59.155142 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" event={"ID":"ed378841-7aa5-4141-852f-b0cfbf784049","Type":"ContainerStarted","Data":"27238231dd67269f40b15e556f61fa9c6956fcda78b6a739abb12b124fa50e3c"} Jul 16 14:22:01.170468 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:01.170427 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" event={"ID":"ed378841-7aa5-4141-852f-b0cfbf784049","Type":"ContainerStarted","Data":"7707af566d0c664d34ce731e8731ecc6edb80ec1d66d12c945f483a896128fbc"} Jul 16 14:22:01.171070 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:01.170484 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" event={"ID":"ed378841-7aa5-4141-852f-b0cfbf784049","Type":"ContainerStarted","Data":"6c08c6eb962c51758f64f2fde83776e8baf175e9dedb17baf9372f8c4e577944"} Jul 16 14:22:01.190083 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:01.190019 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mhhqm/must-gather-rxs4b" podStartSLOduration=1.77103867 podStartE2EDuration="4.189995799s" totalImagesPullingTime="2.418957129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 14:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:22:01.187748297 +0000 UTC m=+2501.135865004" watchObservedRunningTime="2026-07-16 14:22:01.189995799 +0000 UTC m=+2501.138112508" Jul 16 14:22:02.463806 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:02.463759 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n9glr_9d7cc4eb-0314-42c9-ae8d-39152296fbda/global-pull-secret-syncer/0.log" Jul 16 14:22:02.524626 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:02.524590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8f6wm_8489bd65-432f-4e2e-8e58-bd4e2463d7a3/konnectivity-agent/0.log" Jul 16 14:22:02.622265 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:02.622233 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-14.ec2.internal_8976d328a4aa8b80317cf76fe353706f/haproxy/0.log" Jul 16 14:22:07.001632 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.001591 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/extract/0.log" Jul 16 14:22:07.039081 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.039048 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/util/0.log" Jul 16 14:22:07.062389 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.062345 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hjkqq_edd66ebd-8e44-4b6c-afc1-9ae5fedaf924/pull/0.log" Jul 16 14:22:07.099157 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.099128 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/extract/0.log" Jul 16 14:22:07.129696 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.129660 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/util/0.log" Jul 16 14:22:07.157292 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.157264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wk2d7_15be7cdd-1996-4b4e-843f-b2874979801f/pull/0.log" Jul 16 14:22:07.192993 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.192969 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/extract/0.log" Jul 16 14:22:07.220610 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.220575 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/util/0.log" Jul 16 14:22:07.253708 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.250031 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735hwmv_5ce7cafc-c833-4dd7-b566-4d7620d2eb2a/pull/0.log" Jul 16 14:22:07.290899 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.290080 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/extract/0.log" Jul 16 14:22:07.318153 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.318122 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/util/0.log" Jul 16 14:22:07.342124 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.342092 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qm72n_1d99b1fa-1f2f-4346-8918-e74c2208c918/pull/0.log" Jul 16 14:22:07.840896 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.840509 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-564fb575c6-jxlnd_4578272c-787f-420f-8a70-3d5e08fa9bea/authorino/0.log" Jul 16 14:22:07.888511 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.888475 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7dljr_4ebf3195-b824-47e9-8b24-719b02824133/manager/0.log" Jul 16 14:22:07.930741 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.928564 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gzp4h_adc7174e-48ed-485e-b686-58461c6f7c9c/manager/0.log" Jul 16 14:22:07.966032 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:07.965596 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fs8np_70b52ffa-2f77-4ac6-bfab-9fcb27b3d425/kuadrant-console-plugin/0.log" Jul 16 14:22:08.032010 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:08.031975 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gkdd4_e8ba6a14-303c-4ef2-8125-e0b1872b3a62/registry-server/0.log" Jul 16 14:22:08.156059 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:08.155965 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q67hs_3d6c259d-1f12-4506-a925-0d1e77039971/manager/0.log" Jul 16 14:22:08.185896 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:08.185850 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7w8bs_9511c1dd-cfb2-494d-859b-908fa156e867/limitador/0.log" Jul 16 14:22:09.779586 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.779553 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/alertmanager/0.log" Jul 16 14:22:09.821896 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.820902 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/config-reloader/0.log" Jul 16 14:22:09.854983 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.854949 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/kube-rbac-proxy-web/0.log" Jul 16 14:22:09.885551 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.885519 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/kube-rbac-proxy/0.log" Jul 16 14:22:09.922514 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.922480 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/kube-rbac-proxy-metric/0.log" Jul 16 14:22:09.953078 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.953030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/prom-label-proxy/0.log" Jul 16 14:22:09.986738 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:09.986705 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6073e961-6fde-430b-9e8f-9db36e67f622/init-config-reloader/0.log" Jul 16 14:22:10.352253 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.352209 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmjmt_060ead13-48f4-4343-881f-85fbc2575782/node-exporter/0.log" Jul 16 14:22:10.383182 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.383098 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmjmt_060ead13-48f4-4343-881f-85fbc2575782/kube-rbac-proxy/0.log" Jul 16 14:22:10.420197 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.420131 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmjmt_060ead13-48f4-4343-881f-85fbc2575782/init-textfile/0.log" Jul 16 14:22:10.643064 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.643030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pvlqf_4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e/kube-rbac-proxy-main/0.log" Jul 16 14:22:10.677890 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.677743 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pvlqf_4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e/kube-rbac-proxy-self/0.log" Jul 16 14:22:10.709601 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:10.709569 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pvlqf_4e8e4397-f7e1-4fd4-a9cf-6cd6f7d1b42e/openshift-state-metrics/0.log" Jul 16 14:22:11.015673 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.015641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-2xgrn_b84787a6-342d-4c14-a474-2b79d9958ff9/prometheus-operator/0.log" Jul 16 14:22:11.036844 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.036813 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-2xgrn_b84787a6-342d-4c14-a474-2b79d9958ff9/kube-rbac-proxy/0.log" Jul 16 14:22:11.067776 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.067746 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-vldnf_1766adb1-a0f5-4c59-8792-52d4fb47882e/prometheus-operator-admission-webhook/0.log" Jul 16 14:22:11.200426 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.200396 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/thanos-query/0.log" Jul 16 14:22:11.244471 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.244440 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/kube-rbac-proxy-web/0.log" Jul 16 14:22:11.270992 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.270906 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/kube-rbac-proxy/0.log" Jul 16 14:22:11.299647 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.299607 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/prom-label-proxy/0.log" Jul 16 14:22:11.329251 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.329221 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/kube-rbac-proxy-rules/0.log" Jul 16 14:22:11.364502 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:11.364381 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-557855ff85-l5b4d_7774c354-37dd-41b2-8ece-bcbd47bd7c56/kube-rbac-proxy-metrics/0.log" Jul 16 14:22:13.605130 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:13.605050 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d6956dc7c-r5m4d_24ad6f90-c8fa-4a04-a549-ba75e7116c9c/console/0.log" Jul 16 14:22:13.647295 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:13.647256 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-t2566_a40e60d9-fecb-48b2-af40-6d2f4c6002f7/download-server/0.log" Jul 16 14:22:13.936708 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:13.936617 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks"] Jul 16 14:22:13.944666 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:13.944634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:13.950723 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:13.950688 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks"] Jul 16 14:22:14.036369 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.036331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-lib-modules\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.036656 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.036630 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-proc\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.036808 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.036795 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-podres\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.037037 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.037018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-sys\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.037188 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.037173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8dv\" (UniqueName: \"kubernetes.io/projected/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-kube-api-access-hp8dv\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138452 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-podres\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138508 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-sys\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8dv\" (UniqueName: \"kubernetes.io/projected/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-kube-api-access-hp8dv\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138566 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-lib-modules\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-proc\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138636 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138605 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-podres\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138844 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-sys\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138844 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-lib-modules\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.138844 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.138705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-proc\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.153956 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.150909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8dv\" (UniqueName: \"kubernetes.io/projected/dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d-kube-api-access-hp8dv\") pod \"perf-node-gather-daemonset-w9lks\" (UID: \"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d\") " pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.258736 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.258641 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:14.475278 ip-10-0-130-14 kubenswrapper[2563]: W0716 14:22:14.475241 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddfc3a8f7_cf67_4d4c_ac92_8413edf42f4d.slice/crio-d6f8b98d9c67e85ffe381ceb492414be6eed6a7171a043a5d585d7943e705c2a WatchSource:0}: Error finding container d6f8b98d9c67e85ffe381ceb492414be6eed6a7171a043a5d585d7943e705c2a: Status 404 returned error can't find the container with id d6f8b98d9c67e85ffe381ceb492414be6eed6a7171a043a5d585d7943e705c2a Jul 16 14:22:14.481802 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:14.481524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks"] Jul 16 14:22:15.317544 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.317498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" event={"ID":"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d","Type":"ContainerStarted","Data":"5cbe28f7b2776ee2d385c1c285e0f55ff40d62d3228c793bb7fb1955973e042f"} Jul 16 14:22:15.318051 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.317570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" event={"ID":"dfc3a8f7-cf67-4d4c-ac92-8413edf42f4d","Type":"ContainerStarted","Data":"d6f8b98d9c67e85ffe381ceb492414be6eed6a7171a043a5d585d7943e705c2a"} Jul 16 14:22:15.318051 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.317615 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:15.341988 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.341927 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" podStartSLOduration=2.341904784 podStartE2EDuration="2.341904784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 14:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 14:22:15.34013643 +0000 UTC m=+2515.288253131" watchObservedRunningTime="2026-07-16 14:22:15.341904784 +0000 UTC m=+2515.290021490" Jul 16 14:22:15.493089 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.493058 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rrd8j_3488b5d2-1e52-4605-91b3-7a0248cffd18/dns/0.log" Jul 16 14:22:15.517400 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.517375 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rrd8j_3488b5d2-1e52-4605-91b3-7a0248cffd18/kube-rbac-proxy/0.log" Jul 16 14:22:15.653277 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:15.653195 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lb6pg_2ef62460-d86f-49bc-9de3-3023d384570f/dns-node-resolver/0.log" Jul 16 14:22:16.278502 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:16.278452 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqmpw_01a5411a-ae3d-42a2-98cd-926bd4155ac8/node-ca/0.log" Jul 16 14:22:17.427067 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:17.427040 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b75db759-xwgcj_b7204850-f591-4438-8c51-ace01411191a/kube-auth-proxy/0.log" Jul 16 14:22:17.465903 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:17.465858 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/0.log" Jul 16 14:22:17.524172 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:17.524137 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/1.log" Jul 16 14:22:17.625207 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:17.625170 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-wswqx_b6224b78-f7ed-435e-9283-5fcf886e4e5d/payload-processing/0.log" Jul 16 14:22:17.662577 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:17.662543 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5845f855c8-qr8xb_a3383b24-2dfc-45d3-9480-855b6e22b92e/router/0.log" Jul 16 14:22:18.379197 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:18.379159 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-252xq_48a1bdd5-e23f-4f26-947f-61e7965cad0c/serve-healthcheck-canary/0.log" Jul 16 14:22:19.076843 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:19.076815 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gghzs_74c1a604-b97f-4033-919d-6af5f3c9993c/kube-rbac-proxy/0.log" Jul 16 14:22:19.100117 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:19.100042 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gghzs_74c1a604-b97f-4033-919d-6af5f3c9993c/exporter/0.log" Jul 16 14:22:19.123505 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:19.123467 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gghzs_74c1a604-b97f-4033-919d-6af5f3c9993c/extractor/0.log" Jul 16 14:22:21.337897 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:21.337798 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mhhqm/perf-node-gather-daemonset-w9lks" Jul 16 14:22:21.705398 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:21.705320 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kxkwr_7e77e136-2d2c-40a8-b152-7485f4a6339b/server/0.log" Jul 16 14:22:21.820545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:21.820505 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fctwd_bb228b98-3ec5-4802-a27e-8993fa0c68e9/manager/0.log" Jul 16 14:22:21.901926 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:21.901893 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-558d6dc665-7fzrq_c06c9634-bc6c-4670-8311-47245d2731ba/manager/0.log" Jul 16 14:22:21.924344 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:21.924316 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-558d6dc665-7fzrq_c06c9634-bc6c-4670-8311-47245d2731ba/copy-manifests/0.log" Jul 16 14:22:23.693334 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:23.693307 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b75db759-xwgcj_b7204850-f591-4438-8c51-ace01411191a/kube-auth-proxy/0.log" Jul 16 14:22:23.715591 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:23.715556 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/0.log" Jul 16 14:22:23.774897 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:23.774298 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/1.log" Jul 16 14:22:23.827011 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:23.826968 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-wswqx_b6224b78-f7ed-435e-9283-5fcf886e4e5d/payload-processing/0.log" Jul 16 14:22:23.841981 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:23.841951 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5845f855c8-qr8xb_a3383b24-2dfc-45d3-9480-855b6e22b92e/router/0.log" Jul 16 14:22:25.170573 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:25.170545 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b75db759-xwgcj_b7204850-f591-4438-8c51-ace01411191a/kube-auth-proxy/0.log" Jul 16 14:22:25.197958 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:25.197926 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/0.log" Jul 16 14:22:25.260256 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:25.260227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6247_0a0054dc-68ba-4482-a4b6-138f93c75a51/istio-proxy/1.log" Jul 16 14:22:25.312794 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:25.312764 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-wswqx_b6224b78-f7ed-435e-9283-5fcf886e4e5d/payload-processing/0.log" Jul 16 14:22:25.336910 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:25.336865 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5845f855c8-qr8xb_a3383b24-2dfc-45d3-9480-855b6e22b92e/router/0.log" Jul 16 14:22:28.759187 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:28.759157 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7klwt_d790fecd-6593-4af5-8bb5-f5d17280ea1b/migrator/0.log" Jul 16 14:22:28.780019 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:28.779994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7klwt_d790fecd-6593-4af5-8bb5-f5d17280ea1b/graceful-termination/0.log" Jul 16 14:22:30.230965 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.230935 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9c256_9aa7c6c5-3bea-47c6-81b3-fb8d6f2098d3/kube-multus/0.log" Jul 16 14:22:30.274951 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.274918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/kube-multus-additional-cni-plugins/0.log" Jul 16 14:22:30.303609 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.303582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/egress-router-binary-copy/0.log" Jul 16 14:22:30.328545 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.328515 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/cni-plugins/0.log" Jul 16 14:22:30.353710 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.353672 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/bond-cni-plugin/0.log" Jul 16 14:22:30.379055 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.379022 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/routeoverride-cni/0.log" Jul 16 14:22:30.401078 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.401055 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/whereabouts-cni-bincopy/0.log" Jul 16 14:22:30.422786 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.422762 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt66x_70c04af3-de33-48d5-9959-2b6c5fd45515/whereabouts-cni/0.log" Jul 16 14:22:30.958092 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.958063 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kf4n6_8c18e82a-f845-4475-bc78-f29acf434c83/network-metrics-daemon/0.log" Jul 16 14:22:30.978316 ip-10-0-130-14 kubenswrapper[2563]: I0716 14:22:30.978282 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kf4n6_8c18e82a-f845-4475-bc78-f29acf434c83/kube-rbac-proxy/0.log"